2024-04-15T15:04:26.764 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T15:04:26.765 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} -- /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/orig.config.yaml 2024-04-15T15:04:26.811 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:04:26.904 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123 branch: wip-lusov-quiesce-xlock description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} email: null first_in_suite: false job_id: '7657123' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiesce-xlock ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiesce-xlock sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2288 sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 sleep_before_teardown: 0 suite: fs suite_branch: wip-lusov-quiesce-xlock suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9vR3gjx85IDEfOdCoscp48OpEZQiPGteXc/kLsFQoKsj1FtDW56a5VP1VFl1/MZbE3p1Oqz2pcwSST7cMdE0Y= smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWnxiGt5uOpN3O37WMgvVBLr5l137jxAV/h1H3Zhujy7KjZ40v1QMx6J6bXcq6NOBEL8EhbD9c3MroETUQ4HjQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_quiesce teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_14:46:31 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T15:04:26.904 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa; will attempt to use it 2024-04-15T15:04:26.905 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks 2024-04-15T15:04:26.905 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T15:04:26.906 INFO:teuthology.task.internal:Checking packages... 2024-04-15T15:04:26.931 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14' 2024-04-15T15:04:26.931 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T15:04:26.931 INFO:teuthology.packaging:ref: None 2024-04-15T15:04:26.931 INFO:teuthology.packaging:tag: None 2024-04-15T15:04:26.931 INFO:teuthology.packaging:branch: wip-lusov-quiesce-xlock 2024-04-15T15:04:26.931 INFO:teuthology.packaging:sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:04:26.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiesce-xlock 2024-04-15T15:04:27.072 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2861.g9bf1c4ba 2024-04-15T15:04:27.074 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T15:04:27.081 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T15:04:27.081 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T15:04:27.094 INFO:teuthology.task.internal:Saving configuration 2024-04-15T15:04:27.110 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T15:04:27.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:04:27.141 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:18.559692', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9vR3gjx85IDEfOdCoscp48OpEZQiPGteXc/kLsFQoKsj1FtDW56a5VP1VFl1/MZbE3p1Oqz2pcwSST7cMdE0Y='} 2024-04-15T15:04:27.167 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:18.562129', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWnxiGt5uOpN3O37WMgvVBLr5l137jxAV/h1H3Zhujy7KjZ40v1QMx6J6bXcq6NOBEL8EhbD9c3MroETUQ4HjQ='} 2024-04-15T15:04:27.167 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T15:04:27.174 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-15T15:04:27.174 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-15T15:04:27.174 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T15:04:27.249 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9aab7f70d0>, signals=[15]) 2024-04-15T15:04:27.250 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T15:04:27.258 INFO:teuthology.task.internal:Opening connections... 2024-04-15T15:04:27.258 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:04:27.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:04:27.386 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:04:27.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:04:27.475 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T15:04:27.482 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-04-15T15:04:27.500 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-04-15T15:04:27.500 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:04:27.557 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:04:27.558 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:04:27.558 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:04:27.558 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:04:27.558 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:04:27.558 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-04-15T15:04:27.585 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-04-15T15:04:27.603 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-04-15T15:04:27.603 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-04-15T15:04:27.660 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2024-04-15T15:04:27.660 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="9" 2024-04-15T15:04:27.660 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2024-04-15T15:04:27.660 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:04:27.660 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="9" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:04:27.661 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:04:27.661 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-04-15T15:04:27.684 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T15:04:27.694 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T15:04:27.701 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T15:04:27.701 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:04:27.704 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:04:27.721 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T15:04:27.727 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T15:04:27.728 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:04:27.761 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:04:27.870 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T15:04:27.927 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T15:04:27.942 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-15T15:04:27.942 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-15T15:04:27.942 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:04:27.943 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:04:27.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:04:27.960 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-04-15T15:04:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:04:27.961 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-04-15T15:04:28.017 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:04:28.017 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.14.0-361.el9.x86_64 2024-04-15T15:04:28.017 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2024-04-15T15:04:28.019 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:04:28.019 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-361.el9.x86_64 2024-04-15T15:04:28.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-04-15T15:04:30.544 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-04-15T15:04:30.574 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-04-15T15:04:32.247 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-04-15T15:04:32.328 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-04-15T15:04:33.168 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-15T15:04:33.614 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-04-15T15:04:35.111 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-15T15:04:37.906 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-04-15T15:04:39.137 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-15T15:04:40.771 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-15T15:04:41.757 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-15T15:04:41.974 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-15T15:04:43.575 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-15T15:04:44.909 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-15T15:04:46.915 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-15T15:04:49.297 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:04:49.297 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:04:49.387 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-04-15T15:04:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2024-04-15T15:04:49.396 INFO:teuthology.orchestra.run.smithi078.stdout:Install 4 Packages 2024-04-15T15:04:49.396 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:04:49.396 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 77 M 2024-04-15T15:04:49.396 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 123 M 2024-04-15T15:04:49.396 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-04-15T15:04:49.666 INFO:teuthology.orchestra.run.smithi078.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.0 MB/s | 277 kB 00:00 2024-04-15T15:04:50.067 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-15T15:04:50.567 INFO:teuthology.orchestra.run.smithi078.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-04-15T15:04:51.250 INFO:teuthology.orchestra.run.smithi078.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-15T15:04:51.800 INFO:teuthology.orchestra.run.smithi078.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-04-15T15:04:51.804 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:04:51.804 INFO:teuthology.orchestra.run.smithi078.stdout:Total 32 MB/s | 77 MB 00:02 2024-04-15T15:04:51.805 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-04-15T15:04:52.160 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-04-15T15:04:52.160 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-04-15T15:04:52.583 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:04:52.583 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:04:52.683 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-15T15:04:52.687 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-15T15:04:52.687 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-15T15:04:52.687 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-15T15:04:52.687 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-15T15:04:52.687 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-04-15T15:04:52.688 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:04:52.689 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 77 M 2024-04-15T15:04:52.689 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 123 M 2024-04-15T15:04:52.689 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-15T15:04:52.910 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.2 MB/s | 277 kB 00:00 2024-04-15T15:04:53.032 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-04-15T15:04:53.032 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-04-15T15:04:53.669 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-04-15T15:04:54.144 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 22 MB/s | 28 MB 00:01 2024-04-15T15:04:54.511 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 18 MB/s | 34 MB 00:01 2024-04-15T15:04:54.514 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:04:54.514 INFO:teuthology.orchestra.run.smithi046.stdout:Total 42 MB/s | 77 MB 00:01 2024-04-15T15:04:54.515 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-15T15:04:54.945 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-15T15:04:54.945 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-15T15:04:55.336 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-04-15T15:04:55.818 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-15T15:04:55.818 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-15T15:04:55.960 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:04:56.043 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:58.250 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-15T15:04:58.770 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:58.889 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:04:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:58.978 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:01.908 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:02.053 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:03.633 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:03.781 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:07.174 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:07.211 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:08.509 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:37.796 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:37.797 INFO:teuthology.orchestra.run.smithi078.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:05:37.797 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:05:37.842 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:38.227 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:05:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:38.228 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:05:38.626 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-04-15T15:05:38.964 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:39.063 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:39.063 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:39.063 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:05:39.063 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:05:39.063 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:05:39.063 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T15:05:39.064 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:05:39.064 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:39.144 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:39.144 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:39.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2024-04-15T15:05:39.184 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2024-04-15T15:05:39.184 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2024-04-15T15:05:39.184 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.06 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 68.el9 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Group : Unspecified 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 8264254 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:05:39.185 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : builder@centos.org 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:05:39.186 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:05:39.187 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:05:39.187 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:05:39.187 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:05:39.777 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-04-15T15:05:41.128 INFO:teuthology.orchestra.run.smithi078.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:05:41.164 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-04-15T15:05:41.167 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-04-15T15:05:41.182 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.UZUA7zr99d 2024-04-15T15:05:41.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.UZUA7zr99d 2024-04-15T15:05:41.250 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.UZUA7zr99d 2024-04-15T15:05:41.396 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.UZUA7zr99d is 6KB 2024-04-15T15:05:41.444 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.UZUA7zr99d 2024-04-15T15:05:41.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:05:41.532 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:05:41.533 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:05:41.533 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:05:41.533 INFO:teuthology.orchestra.run.smithi078.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:05:41.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:05:41.682 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-04-15T15:05:43.337 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:43.337 INFO:teuthology.orchestra.run.smithi046.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:05:43.337 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:05:43.382 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:43.775 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:43.775 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:05:43.775 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:43.775 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:44.200 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:05:44.201 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-15T15:05:44.511 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:44.623 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:44.623 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:44.624 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:05:44.624 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:05:44.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:05:44.624 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T15:05:44.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:05:44.624 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:44.707 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:44.707 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:44.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-04-15T15:05:44.754 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-04-15T15:05:44.754 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-04-15T15:05:44.754 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-04-15T15:05:44.754 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2024-04-15T15:05:44.754 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:05:44.755 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:05:44.756 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:05:44.756 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:05:44.756 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-04-15T15:05:44.756 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:05:44.756 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:05:44.756 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:05:44.757 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:05:44.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:05:45.396 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-04-15T15:05:46.895 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18064: grub2-probe 2024-04-15T15:05:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18064: grub2-probe 2024-04-15T15:05:46.930 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18064: grub2-probe 2024-04-15T15:05:46.930 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18064: grub2-probe 2024-04-15T15:05:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18112: grub2-probe 2024-04-15T15:05:47.152 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18112: grub2-probe 2024-04-15T15:05:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18112: grub2-probe 2024-04-15T15:05:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18112: grub2-probe 2024-04-15T15:05:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18160: grub2-probe 2024-04-15T15:05:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18160: grub2-probe 2024-04-15T15:05:47.339 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18160: grub2-probe 2024-04-15T15:05:47.339 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18160: grub2-probe 2024-04-15T15:05:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18208: grub2-probe 2024-04-15T15:05:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18208: grub2-probe 2024-04-15T15:05:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18208: grub2-probe 2024-04-15T15:05:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18208: grub2-probe 2024-04-15T15:05:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18256: grub2-probe 2024-04-15T15:05:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18256: grub2-probe 2024-04-15T15:05:47.730 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[26142]) leaked on vgs invocation. Parent PID 18256: grub2-probe 2024-04-15T15:05:47.730 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[42896]) leaked on vgs invocation. Parent PID 18256: grub2-probe 2024-04-15T15:05:47.925 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:05:47.947 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-04-15T15:05:47.948 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-04-15T15:05:47.963 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.y2y8KHNGeq 2024-04-15T15:05:47.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.y2y8KHNGeq 2024-04-15T15:05:48.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.y2y8KHNGeq 2024-04-15T15:05:48.179 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.y2y8KHNGeq is 6KB 2024-04-15T15:05:48.228 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.y2y8KHNGeq 2024-04-15T15:05:48.243 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:05:48.317 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:05:48.317 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:05:48.317 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:05:48.317 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:05:48.318 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:05:48.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-04-15T15:06:11.706 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T15:06:11.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-15T15:06:11.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:18.478 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T15:06:18.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:06:18.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:30.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-04-15T15:06:36.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-04-15T15:06:39.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-15T15:06:39.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:42.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-04-15T15:06:45.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:06:45.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:49.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-04-15T15:06:54.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-15T15:06:54.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:07:01.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:07:01.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:07:25.938 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-04-15T15:07:26.531 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-15T15:07:26.531 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:07:26.531 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-04-15T15:07:26.554 INFO:teuthology.orchestra.run.smithi078.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:07:26.554 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:07:26.554 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:07:26.555 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:07:27.555 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T15:07:27.555 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:07:27.914 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-04-15T15:07:28.027 DEBUG:teuthology.parallel:result is None 2024-04-15T15:08:01.028 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T15:08:16.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:08:16.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:08:16.311 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-04-15T15:08:16.688 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:08:16.688 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:08:16.688 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-04-15T15:08:16.703 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:08:16.703 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:08:16.703 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:08:16.703 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:08:17.704 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T15:08:17.704 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:08:17.794 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-04-15T15:08:17.825 DEBUG:teuthology.parallel:result is None 2024-04-15T15:08:17.825 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T15:08:17.832 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T15:08:17.832 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:08:17.834 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:08:17.853 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T15:08:17.919 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T15:08:17.928 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T15:08:17.929 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:08:17.931 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:08:17.977 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T15:08:17.983 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T15:08:17.984 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:08:17.998 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:08:18.042 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:18.055 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:18.057 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:18.070 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:18.072 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T15:08:18.079 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T15:08:18.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:08:18.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:08:18.143 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T15:08:18.153 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T15:08:18.153 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:08:18.155 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:08:18.199 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:18.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:18.305 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:18.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:18.432 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:08:18.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:08:18.502 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:18.544 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:18.598 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:18.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:18.727 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:08:18.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:08:18.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-04-15T15:08:18.800 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-04-15T15:08:18.847 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:08:18.897 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:08:19.346 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T15:08:19.354 INFO:teuthology.task.internal:Starting timer... 2024-04-15T15:08:19.354 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T15:08:19.378 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T15:08:19.386 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-15T15:08:19.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-04-15T15:08:19.444 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-04-15T15:08:19.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2024-04-15T15:08:19.508 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2024-04-15T15:08:19.510 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T15:08:19.510 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-04-15T15:08:19.540 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-04-15T15:08:19.540 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2024-04-15T15:08:19.560 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2024-04-15T15:08:19.560 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi078.front.sepia.ceph.com': 'permissive'} 2024-04-15T15:08:19.561 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:08:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:08:19.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:08:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:08:19.642 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T15:08:19.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:08:19.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:08:19.737 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T15:08:19.746 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T15:08:19.769 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T15:08:19.770 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi078.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T15:12:47.345 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi078.front.sepia.ceph.com')] 2024-04-15T15:12:47.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:12:47.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:47.447 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-04-15T15:12:47.531 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-15T15:12:47.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-15T15:12:47.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:47.622 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-04-15T15:12:47.702 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-04-15T15:12:47.702 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T15:12:47.713 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T15:12:47.713 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:12:47.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:12:47.716 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:12:47.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:12:47.749 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:12:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:12:47.776 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:12:47.795 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:12:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-04-15T15:12:47.814 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-04-15T15:12:47.826 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2024-04-15T15:12:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:12:47.840 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2024-04-15T15:12:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:12:47.857 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:12:47.874 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:12:47.923 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:12:47.968 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:12:47.968 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-15T15:12:47.968 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.968 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.969 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.969 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.969 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:12:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:12:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-04-15T15:12:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.970 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:47.970 INFO:teuthology.run_tasks:Running task install... 2024-04-15T15:12:47.981 DEBUG:teuthology.task.install:project ceph 2024-04-15T15:12:47.981 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}} 2024-04-15T15:12:47.981 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'} 2024-04-15T15:12:47.981 INFO:teuthology.task.install:Using flavor: default 2024-04-15T15:12:47.987 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-15T15:12:47.987 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-15T15:12:47.988 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:12:47.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:12:47.991 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:12:47.991 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:12:48.137 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:12:48.138 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:12:48.211 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:12:48.211 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:12:48.340 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:12:48.340 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:12:48.340 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:12:48.370 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:12:48.371 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:12:48.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:12:48.483 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:12:48.484 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:12:48.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:12:48.513 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:12:48.513 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:12:48.527 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:12:48.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:12:48.595 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-04-15T15:12:48.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-04-15T15:12:48.664 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:12:48.732 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2024-04-15T15:12:48.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2024-04-15T15:12:48.852 INFO:teuthology.orchestra.run.smithi046.stdout:74 files removed 2024-04-15T15:12:48.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:12:48.975 INFO:teuthology.orchestra.run.smithi078.stdout:74 files removed 2024-04-15T15:12:49.002 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:12:49.845 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 128 kB/s | 83 kB 00:00 2024-04-15T15:12:49.986 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 122 kB/s | 83 kB 00:00 2024-04-15T15:12:50.538 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 18 kB/s | 11 kB 00:00 2024-04-15T15:12:50.569 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 22 kB/s | 11 kB 00:00 2024-04-15T15:12:51.255 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-04-15T15:12:51.270 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-04-15T15:12:51.906 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-15T15:12:51.932 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-15T15:12:53.474 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-04-15T15:12:53.498 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-04-15T15:12:54.247 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-04-15T15:12:55.582 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-04-15T15:12:58.533 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-15T15:12:59.031 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - AppStream 5.1 MB/s | 19 MB 00:03 2024-04-15T15:13:02.659 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-15T15:13:04.452 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-15T15:13:05.519 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-15T15:13:05.915 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-04-15T15:13:08.416 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-04-15T15:13:09.709 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-15T15:13:10.845 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-15T15:13:13.667 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:13:13.668 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:13:13.685 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:13:13.686 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:13:13.686 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:13:13.686 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:13:13.687 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:13:13.690 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:13:13.691 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:13:13.778 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:13:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:13:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:13:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:13:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:13:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:13:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:13:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:13:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:13:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:13:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:13:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:13:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:13:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:13:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:13:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:13:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:13:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:13:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout:Install 150 Packages 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 24 Packages 2024-04-15T15:13:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:13:13.812 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 228 M 2024-04-15T15:13:13.812 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-15T15:13:14.571 INFO:teuthology.orchestra.run.smithi046.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-15T15:13:14.725 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-15T15:13:15.129 INFO:teuthology.orchestra.run.smithi046.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.6 MB/s | 890 kB 00:00 2024-04-15T15:13:15.246 INFO:teuthology.orchestra.run.smithi046.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 1.2 MB/s | 145 kB 00:00 2024-04-15T15:13:15.605 INFO:teuthology.orchestra.run.smithi046.stdout:(4/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 4.1 MB/s | 5.4 MB 00:01 2024-04-15T15:13:15.822 INFO:teuthology.orchestra.run.smithi046.stdout:(5/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 4.1 MB/s | 2.4 MB 00:00 2024-04-15T15:13:15.897 INFO:teuthology.orchestra.run.smithi046.stdout:(6/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 5.1 MB/s | 1.5 MB 00:00 2024-04-15T15:13:16.447 INFO:teuthology.orchestra.run.smithi046.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 7.5 MB/s | 4.7 MB 00:00 2024-04-15T15:13:17.231 INFO:teuthology.orchestra.run.smithi046.stdout:(8/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el8 13 MB/s | 10 MB 00:00 2024-04-15T15:13:17.297 INFO:teuthology.orchestra.run.smithi046.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 378 kB/s | 25 kB 00:00 2024-04-15T15:13:17.447 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:13:17.448 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:13:17.462 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:13:17.463 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:13:17.464 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:13:17.465 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:13:17.467 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:13:17.470 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:13:17.471 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:13:17.551 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2024-04-15T15:13:17.574 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-04-15T15:13:17.574 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2024-04-15T15:13:17.574 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-04-15T15:13:17.574 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:13:17.575 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:13:17.577 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:13:17.578 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:13:17.579 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:13:17.580 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:13:17.581 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:13:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:13:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:13:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:13:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:13:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:13:17.582 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:13:17.583 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:13:17.584 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:13:17.584 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:13:17.584 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:13:17.584 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:13:17.584 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:13:17.584 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:13:17.585 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:13:17.586 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:13:17.587 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:13:17.588 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:13:17.589 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:13:17.590 INFO:teuthology.orchestra.run.smithi078.stdout:Installing weak dependencies: 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================ 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout:Install 150 Packages 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 24 Packages 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:13:17.591 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 228 M 2024-04-15T15:13:17.592 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2024-04-15T15:13:18.156 INFO:teuthology.orchestra.run.smithi046.stdout:(10/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8 5.4 MB/s | 21 MB 00:03 2024-04-15T15:13:18.223 INFO:teuthology.orchestra.run.smithi046.stdout:(11/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 3.3 MB/s | 224 kB 00:00 2024-04-15T15:13:18.315 INFO:teuthology.orchestra.run.smithi046.stdout:(12/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 295 kB/s | 27 kB 00:00 2024-04-15T15:13:18.415 INFO:teuthology.orchestra.run.smithi046.stdout:(13/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 317 kB/s | 32 kB 00:00 2024-04-15T15:13:18.518 INFO:teuthology.orchestra.run.smithi078.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-15T15:13:18.682 INFO:teuthology.orchestra.run.smithi046.stdout:(14/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 2.7 MB/s | 747 kB 00:00 2024-04-15T15:13:18.782 INFO:teuthology.orchestra.run.smithi046.stdout:(15/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 1.6 MB/s | 164 kB 00:00 2024-04-15T15:13:18.991 INFO:teuthology.orchestra.run.smithi046.stdout:(16/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 5.4 MB/s | 17 MB 00:03 2024-04-15T15:13:19.016 INFO:teuthology.orchestra.run.smithi046.stdout:(17/174): librados-devel-19.0.0-2861.g9bf1c4ba. 544 kB/s | 127 kB 00:00 2024-04-15T15:13:19.150 INFO:teuthology.orchestra.run.smithi046.stdout:(18/174): libradosstriper1-19.0.0-2861.g9bf1c4b 3.1 MB/s | 501 kB 00:00 2024-04-15T15:13:19.250 INFO:teuthology.orchestra.run.smithi046.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 452 kB/s | 45 kB 00:00 2024-04-15T15:13:19.302 INFO:teuthology.orchestra.run.smithi078.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.1 MB/s | 890 kB 00:00 2024-04-15T15:13:19.350 INFO:teuthology.orchestra.run.smithi046.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 1.3 MB/s | 135 kB 00:00 2024-04-15T15:13:19.459 INFO:teuthology.orchestra.run.smithi046.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.5 MB/s | 162 kB 00:00 2024-04-15T15:13:19.494 INFO:teuthology.orchestra.run.smithi078.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 757 kB/s | 145 kB 00:00 2024-04-15T15:13:19.651 INFO:teuthology.orchestra.run.smithi046.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 1.6 MB/s | 320 kB 00:00 2024-04-15T15:13:20.159 INFO:teuthology.orchestra.run.smithi046.stdout:(23/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8.x 17 MB/s | 49 MB 00:02 2024-04-15T15:13:20.185 INFO:teuthology.orchestra.run.smithi046.stdout:(24/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 557 kB/s | 297 kB 00:00 2024-04-15T15:13:20.219 INFO:teuthology.orchestra.run.smithi078.stdout:(4/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 3.3 MB/s | 2.4 MB 00:00 2024-04-15T15:13:20.260 INFO:teuthology.orchestra.run.smithi046.stdout:(25/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 4.2 MB/s | 5.2 MB 00:01 2024-04-15T15:13:20.285 INFO:teuthology.orchestra.run.smithi046.stdout:(26/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 850 kB/s | 85 kB 00:00 2024-04-15T15:13:20.311 INFO:teuthology.orchestra.run.smithi046.stdout:(27/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 658 kB/s | 99 kB 00:00 2024-04-15T15:13:20.344 INFO:teuthology.orchestra.run.smithi046.stdout:(28/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 2.8 MB/s | 171 kB 00:00 2024-04-15T15:13:20.378 INFO:teuthology.orchestra.run.smithi046.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2861.g 511 kB/s | 34 kB 00:00 2024-04-15T15:13:20.411 INFO:teuthology.orchestra.run.smithi046.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 2.1 MB/s | 142 kB 00:00 2024-04-15T15:13:20.552 INFO:teuthology.orchestra.run.smithi078.stdout:(5/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 4.5 MB/s | 1.5 MB 00:00 2024-04-15T15:13:21.037 INFO:teuthology.orchestra.run.smithi046.stdout:(31/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 4.0 MB/s | 3.1 MB 00:00 2024-04-15T15:13:21.179 INFO:teuthology.orchestra.run.smithi046.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 9.6 MB/s | 7.4 MB 00:00 2024-04-15T15:13:21.204 INFO:teuthology.orchestra.run.smithi046.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2861.g9b 1.5 MB/s | 249 kB 00:00 2024-04-15T15:13:21.229 INFO:teuthology.orchestra.run.smithi046.stdout:(34/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 986 kB/s | 49 kB 00:00 2024-04-15T15:13:21.254 INFO:teuthology.orchestra.run.smithi046.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2861.g9 329 kB/s | 16 kB 00:00 2024-04-15T15:13:21.297 INFO:teuthology.orchestra.run.smithi046.stdout:(36/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 3.9 MB/s | 264 kB 00:00 2024-04-15T15:13:21.388 INFO:teuthology.orchestra.run.smithi046.stdout:(37/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 3.5 MB/s | 3.5 MB 00:01 2024-04-15T15:13:21.405 INFO:teuthology.orchestra.run.smithi046.stdout:(38/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 168 kB/s | 18 kB 00:00 2024-04-15T15:13:21.447 INFO:teuthology.orchestra.run.smithi046.stdout:(39/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 3.9 MB/s | 762 kB 00:00 2024-04-15T15:13:21.630 INFO:teuthology.orchestra.run.smithi046.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 813 kB/s | 106 kB 00:00 2024-04-15T15:13:21.656 INFO:teuthology.orchestra.run.smithi046.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-04-15T15:13:21.669 INFO:teuthology.orchestra.run.smithi078.stdout:(6/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-15T15:13:21.682 INFO:teuthology.orchestra.run.smithi046.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 920 kB 00:00 2024-04-15T15:13:21.715 INFO:teuthology.orchestra.run.smithi046.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-04-15T15:13:21.732 INFO:teuthology.orchestra.run.smithi046.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 423 kB/s | 32 kB 00:00 2024-04-15T15:13:21.736 INFO:teuthology.orchestra.run.smithi078.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 4.0 MB/s | 4.7 MB 00:01 2024-04-15T15:13:21.799 INFO:teuthology.orchestra.run.smithi046.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 206 kB/s | 17 kB 00:00 2024-04-15T15:13:21.899 INFO:teuthology.orchestra.run.smithi046.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-04-15T15:13:21.924 INFO:teuthology.orchestra.run.smithi046.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 52 kB/s | 10 kB 00:00 2024-04-15T15:13:21.949 INFO:teuthology.orchestra.run.smithi046.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-04-15T15:13:21.974 INFO:teuthology.orchestra.run.smithi046.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-04-15T15:13:21.999 INFO:teuthology.orchestra.run.smithi046.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-04-15T15:13:22.025 INFO:teuthology.orchestra.run.smithi046.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-04-15T15:13:22.050 INFO:teuthology.orchestra.run.smithi046.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-04-15T15:13:22.075 INFO:teuthology.orchestra.run.smithi046.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-04-15T15:13:22.101 INFO:teuthology.orchestra.run.smithi046.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-04-15T15:13:22.126 INFO:teuthology.orchestra.run.smithi046.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-04-15T15:13:22.151 INFO:teuthology.orchestra.run.smithi046.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 455 kB/s | 34 kB 00:00 2024-04-15T15:13:22.268 INFO:teuthology.orchestra.run.smithi046.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-04-15T15:13:22.310 INFO:teuthology.orchestra.run.smithi046.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 9.7 MB/s | 1.5 MB 00:00 2024-04-15T15:13:22.344 INFO:teuthology.orchestra.run.smithi046.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 4.2 MB/s | 817 kB 00:00 2024-04-15T15:13:22.369 INFO:teuthology.orchestra.run.smithi046.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 132 kB 00:00 2024-04-15T15:13:22.444 INFO:teuthology.orchestra.run.smithi046.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-15T15:13:22.519 INFO:teuthology.orchestra.run.smithi046.stdout:(62/174): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-04-15T15:13:22.545 INFO:teuthology.orchestra.run.smithi046.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 449 kB/s | 78 kB 00:00 2024-04-15T15:13:22.645 INFO:teuthology.orchestra.run.smithi046.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-04-15T15:13:22.670 INFO:teuthology.orchestra.run.smithi046.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 148 kB/s | 18 kB 00:00 2024-04-15T15:13:22.695 INFO:teuthology.orchestra.run.smithi046.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 165 kB/s | 24 kB 00:00 2024-04-15T15:13:22.721 INFO:teuthology.orchestra.run.smithi046.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-04-15T15:13:22.746 INFO:teuthology.orchestra.run.smithi046.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-04-15T15:13:22.796 INFO:teuthology.orchestra.run.smithi046.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.8 MB/s | 217 kB 00:00 2024-04-15T15:13:22.863 INFO:teuthology.orchestra.run.smithi046.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-04-15T15:13:22.888 INFO:teuthology.orchestra.run.smithi046.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-04-15T15:13:22.913 INFO:teuthology.orchestra.run.smithi046.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 598 kB/s | 69 kB 00:00 2024-04-15T15:13:22.938 INFO:teuthology.orchestra.run.smithi046.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-04-15T15:13:22.955 INFO:teuthology.orchestra.run.smithi046.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 617 kB/s | 41 kB 00:00 2024-04-15T15:13:22.980 INFO:teuthology.orchestra.run.smithi046.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-04-15T15:13:23.006 INFO:teuthology.orchestra.run.smithi046.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-04-15T15:13:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-15T15:13:23.040 INFO:teuthology.orchestra.run.smithi046.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 733 kB/s | 43 kB 00:00 2024-04-15T15:13:23.190 INFO:teuthology.orchestra.run.smithi046.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 262 kB/s | 39 kB 00:00 2024-04-15T15:13:23.207 INFO:teuthology.orchestra.run.smithi046.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 250 kB/s | 41 kB 00:00 2024-04-15T15:13:23.357 INFO:teuthology.orchestra.run.smithi046.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-15T15:13:23.382 INFO:teuthology.orchestra.run.smithi046.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 220 kB 00:00 2024-04-15T15:13:23.407 INFO:teuthology.orchestra.run.smithi046.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 175 kB/s | 35 kB 00:00 2024-04-15T15:13:23.441 INFO:teuthology.orchestra.run.smithi046.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-04-15T15:13:23.466 INFO:teuthology.orchestra.run.smithi046.stdout:(85/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 554 kB/s | 46 kB 00:00 2024-04-15T15:13:23.491 INFO:teuthology.orchestra.run.smithi046.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 678 kB/s | 56 kB 00:00 2024-04-15T15:13:23.517 INFO:teuthology.orchestra.run.smithi046.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 833 kB/s | 63 kB 00:00 2024-04-15T15:13:23.542 INFO:teuthology.orchestra.run.smithi046.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-04-15T15:13:23.567 INFO:teuthology.orchestra.run.smithi046.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-04-15T15:13:23.592 INFO:teuthology.orchestra.run.smithi046.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-15T15:13:23.626 INFO:teuthology.orchestra.run.smithi046.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-04-15T15:13:23.651 INFO:teuthology.orchestra.run.smithi046.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-04-15T15:13:23.676 INFO:teuthology.orchestra.run.smithi046.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-04-15T15:13:23.702 INFO:teuthology.orchestra.run.smithi046.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-04-15T15:13:23.835 INFO:teuthology.orchestra.run.smithi046.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-04-15T15:13:23.861 INFO:teuthology.orchestra.run.smithi046.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-15T15:13:23.886 INFO:teuthology.orchestra.run.smithi046.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-15T15:13:23.886 INFO:teuthology.orchestra.run.smithi078.stdout:(8/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el8 4.6 MB/s | 10 MB 00:02 2024-04-15T15:13:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-15T15:13:23.945 INFO:teuthology.orchestra.run.smithi078.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 432 kB/s | 25 kB 00:00 2024-04-15T15:13:24.045 INFO:teuthology.orchestra.run.smithi046.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 161 kB/s | 25 kB 00:00 2024-04-15T15:13:24.070 INFO:teuthology.orchestra.run.smithi046.stdout:(100/174): elfutils-libelf-devel-0.190-2.el9.x8 103 kB/s | 21 kB 00:00 2024-04-15T15:13:24.287 INFO:teuthology.orchestra.run.smithi046.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.0 MB/s | 245 kB 00:00 2024-04-15T15:13:24.312 INFO:teuthology.orchestra.run.smithi046.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.0 MB/s | 248 kB 00:00 2024-04-15T15:13:24.387 INFO:teuthology.orchestra.run.smithi046.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 936 kB/s | 94 kB 00:00 2024-04-15T15:13:24.413 INFO:teuthology.orchestra.run.smithi046.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-04-15T15:13:24.738 INFO:teuthology.orchestra.run.smithi046.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.4 MB/s | 863 kB 00:00 2024-04-15T15:13:24.914 INFO:teuthology.orchestra.run.smithi046.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.2 MB/s | 4.1 MB 00:00 2024-04-15T15:13:25.071 INFO:teuthology.orchestra.run.smithi078.stdout:(10/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8 3.0 MB/s | 21 MB 00:06 2024-04-15T15:13:25.122 INFO:teuthology.orchestra.run.smithi046.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-04-15T15:13:25.145 INFO:teuthology.orchestra.run.smithi078.stdout:(11/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 2.9 MB/s | 224 kB 00:00 2024-04-15T15:13:25.196 INFO:teuthology.orchestra.run.smithi078.stdout:(12/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 541 kB/s | 27 kB 00:00 2024-04-15T15:13:25.246 INFO:teuthology.orchestra.run.smithi078.stdout:(13/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 637 kB/s | 32 kB 00:00 2024-04-15T15:13:25.264 INFO:teuthology.orchestra.run.smithi046.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 56 kB/s | 19 kB 00:00 2024-04-15T15:13:25.364 INFO:teuthology.orchestra.run.smithi046.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 494 kB/s | 309 kB 00:00 2024-04-15T15:13:25.389 INFO:teuthology.orchestra.run.smithi046.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-04-15T15:13:25.473 INFO:teuthology.orchestra.run.smithi046.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-15T15:13:25.522 INFO:teuthology.orchestra.run.smithi078.stdout:(14/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 4.4 MB/s | 17 MB 00:03 2024-04-15T15:13:25.540 INFO:teuthology.orchestra.run.smithi046.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-15T15:13:25.546 INFO:teuthology.orchestra.run.smithi078.stdout:(15/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 2.4 MB/s | 747 kB 00:00 2024-04-15T15:13:25.565 INFO:teuthology.orchestra.run.smithi046.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 384 kB/s | 67 kB 00:00 2024-04-15T15:13:25.588 INFO:teuthology.orchestra.run.smithi078.stdout:(16/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 2.4 MB/s | 164 kB 00:00 2024-04-15T15:13:25.599 INFO:teuthology.orchestra.run.smithi046.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-15T15:13:25.622 INFO:teuthology.orchestra.run.smithi078.stdout:(17/174): librados-devel-19.0.0-2861.g9bf1c4ba. 1.6 MB/s | 127 kB 00:00 2024-04-15T15:13:25.649 INFO:teuthology.orchestra.run.smithi046.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-04-15T15:13:25.708 INFO:teuthology.orchestra.run.smithi046.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-04-15T15:13:25.722 INFO:teuthology.orchestra.run.smithi078.stdout:(18/174): libradosstriper1-19.0.0-2861.g9bf1c4b 3.7 MB/s | 501 kB 00:00 2024-04-15T15:13:25.749 INFO:teuthology.orchestra.run.smithi046.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-15T15:13:25.773 INFO:teuthology.orchestra.run.smithi078.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 901 kB/s | 45 kB 00:00 2024-04-15T15:13:25.808 INFO:teuthology.orchestra.run.smithi046.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-15T15:13:25.831 INFO:teuthology.orchestra.run.smithi078.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 2.3 MB/s | 135 kB 00:00 2024-04-15T15:13:25.850 INFO:teuthology.orchestra.run.smithi046.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-15T15:13:25.875 INFO:teuthology.orchestra.run.smithi046.stdout:(120/174): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-04-15T15:13:25.890 INFO:teuthology.orchestra.run.smithi078.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 2.7 MB/s | 162 kB 00:00 2024-04-15T15:13:25.950 INFO:teuthology.orchestra.run.smithi046.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-15T15:13:25.965 INFO:teuthology.orchestra.run.smithi078.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 4.2 MB/s | 320 kB 00:00 2024-04-15T15:13:25.970 INFO:teuthology.orchestra.run.smithi046.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 384 kB/s | 35 kB 00:00 2024-04-15T15:13:25.993 INFO:teuthology.orchestra.run.smithi046.stdout:(123/174): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-04-15T15:13:26.032 INFO:teuthology.orchestra.run.smithi078.stdout:(23/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 4.4 MB/s | 297 kB 00:00 2024-04-15T15:13:26.051 INFO:teuthology.orchestra.run.smithi046.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-04-15T15:13:26.082 INFO:teuthology.orchestra.run.smithi078.stdout:(24/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 1.9 MB/s | 99 kB 00:00 2024-04-15T15:13:26.110 INFO:teuthology.orchestra.run.smithi046.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-04-15T15:13:26.133 INFO:teuthology.orchestra.run.smithi078.stdout:(25/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.7 MB/s | 85 kB 00:00 2024-04-15T15:13:26.161 INFO:teuthology.orchestra.run.smithi046.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-15T15:13:26.211 INFO:teuthology.orchestra.run.smithi046.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-15T15:13:26.269 INFO:teuthology.orchestra.run.smithi046.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.8 MB/s | 2.0 MB 00:00 2024-04-15T15:13:26.294 INFO:teuthology.orchestra.run.smithi046.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-04-15T15:13:26.320 INFO:teuthology.orchestra.run.smithi046.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-04-15T15:13:26.362 INFO:teuthology.orchestra.run.smithi046.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-04-15T15:13:26.420 INFO:teuthology.orchestra.run.smithi046.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-04-15T15:13:26.445 INFO:teuthology.orchestra.run.smithi046.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-04-15T15:13:26.458 INFO:teuthology.orchestra.run.smithi078.stdout:(26/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 6.3 MB/s | 5.2 MB 00:00 2024-04-15T15:13:26.470 INFO:teuthology.orchestra.run.smithi046.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 726 kB/s | 79 kB 00:00 2024-04-15T15:13:26.521 INFO:teuthology.orchestra.run.smithi046.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-04-15T15:13:26.525 INFO:teuthology.orchestra.run.smithi078.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 2.5 MB/s | 171 kB 00:00 2024-04-15T15:13:26.571 INFO:teuthology.orchestra.run.smithi046.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-15T15:13:26.575 INFO:teuthology.orchestra.run.smithi078.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 687 kB/s | 34 kB 00:00 2024-04-15T15:13:26.596 INFO:teuthology.orchestra.run.smithi046.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-15T15:13:26.634 INFO:teuthology.orchestra.run.smithi078.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 2.4 MB/s | 142 kB 00:00 2024-04-15T15:13:26.684 INFO:teuthology.orchestra.run.smithi078.stdout:(30/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 5.6 MB/s | 3.1 MB 00:00 2024-04-15T15:13:26.697 INFO:teuthology.orchestra.run.smithi046.stdout:(138/174): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-04-15T15:13:26.739 INFO:teuthology.orchestra.run.smithi046.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 169 kB/s | 28 kB 00:00 2024-04-15T15:13:26.764 INFO:teuthology.orchestra.run.smithi046.stdout:(140/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 371 kB/s | 90 kB 00:00 2024-04-15T15:13:26.823 INFO:teuthology.orchestra.run.smithi046.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-15T15:13:26.848 INFO:teuthology.orchestra.run.smithi046.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-04-15T15:13:26.873 INFO:teuthology.orchestra.run.smithi046.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-15T15:13:26.923 INFO:teuthology.orchestra.run.smithi046.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-04-15T15:13:26.948 INFO:teuthology.orchestra.run.smithi046.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-15T15:13:26.974 INFO:teuthology.orchestra.run.smithi046.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 892 kB/s | 90 kB 00:00 2024-04-15T15:13:27.049 INFO:teuthology.orchestra.run.smithi046.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-15T15:13:27.074 INFO:teuthology.orchestra.run.smithi046.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-15T15:13:27.099 INFO:teuthology.orchestra.run.smithi046.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-04-15T15:13:27.126 INFO:teuthology.orchestra.run.smithi078.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 7.2 MB/s | 3.5 MB 00:00 2024-04-15T15:13:27.193 INFO:teuthology.orchestra.run.smithi078.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2861.g9b 3.7 MB/s | 249 kB 00:00 2024-04-15T15:13:27.200 INFO:teuthology.orchestra.run.smithi046.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-15T15:13:27.244 INFO:teuthology.orchestra.run.smithi078.stdout:(33/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 992 kB/s | 49 kB 00:00 2024-04-15T15:13:27.294 INFO:teuthology.orchestra.run.smithi078.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2861.g9 329 kB/s | 16 kB 00:00 2024-04-15T15:13:27.377 INFO:teuthology.orchestra.run.smithi078.stdout:(35/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 3.1 MB/s | 264 kB 00:00 2024-04-15T15:13:27.392 INFO:teuthology.orchestra.run.smithi046.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-04-15T15:13:27.467 INFO:teuthology.orchestra.run.smithi046.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-04-15T15:13:27.501 INFO:teuthology.orchestra.run.smithi046.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-04-15T15:13:27.534 INFO:teuthology.orchestra.run.smithi046.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 7.3 MB/s | 248 kB 00:00 2024-04-15T15:13:27.536 INFO:teuthology.orchestra.run.smithi078.stdout:(36/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 4.7 MB/s | 762 kB 00:00 2024-04-15T15:13:27.576 INFO:teuthology.orchestra.run.smithi046.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-04-15T15:13:27.594 INFO:teuthology.orchestra.run.smithi078.stdout:(37/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 313 kB/s | 18 kB 00:00 2024-04-15T15:13:27.660 INFO:teuthology.orchestra.run.smithi046.stdout:(156/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 5.8 MB/s | 3.4 MB 00:00 2024-04-15T15:13:27.685 INFO:teuthology.orchestra.run.smithi046.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 71 kB 00:00 2024-04-15T15:13:27.760 INFO:teuthology.orchestra.run.smithi046.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 362 kB/s | 36 kB 00:00 2024-04-15T15:13:27.770 INFO:teuthology.orchestra.run.smithi078.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 6.8 MB/s | 7.4 MB 00:01 2024-04-15T15:13:27.786 INFO:teuthology.orchestra.run.smithi046.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-04-15T15:13:27.836 INFO:teuthology.orchestra.run.smithi046.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 460 kB/s | 23 kB 00:00 2024-04-15T15:13:27.861 INFO:teuthology.orchestra.run.smithi046.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-04-15T15:13:27.887 INFO:teuthology.orchestra.run.smithi046.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-04-15T15:13:27.912 INFO:teuthology.orchestra.run.smithi078.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 662 kB 00:00 2024-04-15T15:13:27.912 INFO:teuthology.orchestra.run.smithi046.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-04-15T15:13:27.937 INFO:teuthology.orchestra.run.smithi046.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-04-15T15:13:27.954 INFO:teuthology.orchestra.run.smithi046.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-04-15T15:13:27.971 INFO:teuthology.orchestra.run.smithi046.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 859 kB/s | 29 kB 00:00 2024-04-15T15:13:28.037 INFO:teuthology.orchestra.run.smithi078.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-04-15T15:13:28.054 INFO:teuthology.orchestra.run.smithi046.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2024-04-15T15:13:28.071 INFO:teuthology.orchestra.run.smithi078.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 669 kB/s | 106 kB 00:00 2024-04-15T15:13:28.104 INFO:teuthology.orchestra.run.smithi078.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-04-15T15:13:28.129 INFO:teuthology.orchestra.run.smithi078.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 546 kB/s | 32 kB 00:00 2024-04-15T15:13:28.146 INFO:teuthology.orchestra.run.smithi046.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-04-15T15:13:28.197 INFO:teuthology.orchestra.run.smithi046.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 16 MB/s | 2.3 MB 00:00 2024-04-15T15:13:28.222 INFO:teuthology.orchestra.run.smithi046.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-04-15T15:13:28.247 INFO:teuthology.orchestra.run.smithi046.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 621 kB/s | 31 kB 00:00 2024-04-15T15:13:28.281 INFO:teuthology.orchestra.run.smithi046.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-04-15T15:13:28.296 INFO:teuthology.orchestra.run.smithi078.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 103 kB/s | 17 kB 00:00 2024-04-15T15:13:28.347 INFO:teuthology.orchestra.run.smithi078.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-15T15:13:28.364 INFO:teuthology.orchestra.run.smithi046.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 636 kB/s | 74 kB 00:00 2024-04-15T15:13:28.372 INFO:teuthology.orchestra.run.smithi078.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 135 kB/s | 10 kB 00:00 2024-04-15T15:13:28.397 INFO:teuthology.orchestra.run.smithi078.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-15T15:13:28.414 INFO:teuthology.orchestra.run.smithi078.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-04-15T15:13:28.456 INFO:teuthology.orchestra.run.smithi078.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-15T15:13:28.490 INFO:teuthology.orchestra.run.smithi078.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 578 kB 00:00 2024-04-15T15:13:28.507 INFO:teuthology.orchestra.run.smithi078.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 46 kB 00:00 2024-04-15T15:13:28.549 INFO:teuthology.orchestra.run.smithi078.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-04-15T15:13:28.574 INFO:teuthology.orchestra.run.smithi078.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-04-15T15:13:28.599 INFO:teuthology.orchestra.run.smithi078.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-04-15T15:13:28.616 INFO:teuthology.orchestra.run.smithi078.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 825 kB/s | 34 kB 00:00 2024-04-15T15:13:28.665 INFO:teuthology.orchestra.run.smithi046.stdout:(174/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 2.0 MB/s | 3.1 MB 00:01 2024-04-15T15:13:28.672 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:13:28.672 INFO:teuthology.orchestra.run.smithi046.stdout:Total 15 MB/s | 228 MB 00:14 2024-04-15T15:13:28.792 INFO:teuthology.orchestra.run.smithi078.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-04-15T15:13:28.833 INFO:teuthology.orchestra.run.smithi078.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 7.1 MB/s | 1.5 MB 00:00 2024-04-15T15:13:28.867 INFO:teuthology.orchestra.run.smithi078.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 817 kB 00:00 2024-04-15T15:13:28.901 INFO:teuthology.orchestra.run.smithi078.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 132 kB 00:00 2024-04-15T15:13:29.043 INFO:teuthology.orchestra.run.smithi078.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-04-15T15:13:29.054 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-15T15:13:29.077 INFO:teuthology.orchestra.run.smithi078.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-04-15T15:13:29.102 INFO:teuthology.orchestra.run.smithi078.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-15T15:13:29.227 INFO:teuthology.orchestra.run.smithi078.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-04-15T15:13:29.244 INFO:teuthology.orchestra.run.smithi078.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 176 kB/s | 24 kB 00:00 2024-04-15T15:13:29.261 INFO:teuthology.orchestra.run.smithi078.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 542 kB/s | 18 kB 00:00 2024-04-15T15:13:29.287 INFO:teuthology.orchestra.run.smithi078.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-15T15:13:29.311 INFO:teuthology.orchestra.run.smithi078.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-04-15T15:13:29.349 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-15T15:13:29.349 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-15T15:13:29.462 INFO:teuthology.orchestra.run.smithi078.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-04-15T15:13:29.487 INFO:teuthology.orchestra.run.smithi078.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-04-15T15:13:29.513 INFO:teuthology.orchestra.run.smithi078.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-04-15T15:13:29.538 INFO:teuthology.orchestra.run.smithi078.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-04-15T15:13:29.563 INFO:teuthology.orchestra.run.smithi078.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-04-15T15:13:29.588 INFO:teuthology.orchestra.run.smithi078.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi078.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-04-15T15:13:29.639 INFO:teuthology.orchestra.run.smithi078.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-04-15T15:13:29.664 INFO:teuthology.orchestra.run.smithi078.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-04-15T15:13:29.689 INFO:teuthology.orchestra.run.smithi078.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-04-15T15:13:29.998 INFO:teuthology.orchestra.run.smithi078.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-04-15T15:13:30.023 INFO:teuthology.orchestra.run.smithi078.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 117 kB/s | 39 kB 00:00 2024-04-15T15:13:30.049 INFO:teuthology.orchestra.run.smithi078.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 820 kB/s | 41 kB 00:00 2024-04-15T15:13:30.074 INFO:teuthology.orchestra.run.smithi078.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-15T15:13:30.099 INFO:teuthology.orchestra.run.smithi078.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 698 kB/s | 35 kB 00:00 2024-04-15T15:13:30.125 INFO:teuthology.orchestra.run.smithi078.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-04-15T15:13:30.150 INFO:teuthology.orchestra.run.smithi078.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-04-15T15:13:30.167 INFO:teuthology.orchestra.run.smithi078.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-04-15T15:13:30.183 INFO:teuthology.orchestra.run.smithi078.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-04-15T15:13:30.201 INFO:teuthology.orchestra.run.smithi078.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-15T15:13:30.226 INFO:teuthology.orchestra.run.smithi078.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-04-15T15:13:30.251 INFO:teuthology.orchestra.run.smithi078.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-15T15:13:30.285 INFO:teuthology.orchestra.run.smithi078.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-04-15T15:13:30.310 INFO:teuthology.orchestra.run.smithi078.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-04-15T15:13:30.336 INFO:teuthology.orchestra.run.smithi078.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-04-15T15:13:30.361 INFO:teuthology.orchestra.run.smithi078.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-04-15T15:13:30.503 INFO:teuthology.orchestra.run.smithi078.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-15T15:13:30.553 INFO:teuthology.orchestra.run.smithi078.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-15T15:13:30.587 INFO:teuthology.orchestra.run.smithi078.stdout:(96/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-15T15:13:30.612 INFO:teuthology.orchestra.run.smithi078.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-15T15:13:30.762 INFO:teuthology.orchestra.run.smithi078.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 123 kB/s | 21 kB 00:00 2024-04-15T15:13:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 153 kB/s | 25 kB 00:00 2024-04-15T15:13:30.955 INFO:teuthology.orchestra.run.smithi078.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.4 MB/s | 245 kB 00:00 2024-04-15T15:13:31.030 INFO:teuthology.orchestra.run.smithi078.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 3.2 MB/s | 248 kB 00:00 2024-04-15T15:13:31.097 INFO:teuthology.orchestra.run.smithi078.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-04-15T15:13:31.272 INFO:teuthology.orchestra.run.smithi078.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-04-15T15:13:31.422 INFO:teuthology.orchestra.run.smithi078.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.6 MB/s | 863 kB 00:00 2024-04-15T15:13:31.454 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-15T15:13:31.455 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-15T15:13:31.539 INFO:teuthology.orchestra.run.smithi078.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 5.4 MB/s | 4.1 MB 00:00 2024-04-15T15:13:32.290 INFO:teuthology.orchestra.run.smithi078.stdout:(106/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8. 5.8 MB/s | 49 MB 00:08 2024-04-15T15:13:32.323 INFO:teuthology.orchestra.run.smithi078.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 394 kB/s | 309 kB 00:00 2024-04-15T15:13:32.390 INFO:teuthology.orchestra.run.smithi078.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 566 kB/s | 548 kB 00:00 2024-04-15T15:13:32.482 INFO:teuthology.orchestra.run.smithi078.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-15T15:13:32.516 INFO:teuthology.orchestra.run.smithi078.stdout:(110/174): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-04-15T15:13:32.574 INFO:teuthology.orchestra.run.smithi078.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-04-15T15:13:32.674 INFO:teuthology.orchestra.run.smithi078.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-15T15:13:32.699 INFO:teuthology.orchestra.run.smithi078.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-04-15T15:13:32.766 INFO:teuthology.orchestra.run.smithi078.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-15T15:13:32.858 INFO:teuthology.orchestra.run.smithi078.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-04-15T15:13:32.884 INFO:teuthology.orchestra.run.smithi078.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-04-15T15:13:32.951 INFO:teuthology.orchestra.run.smithi078.stdout:(117/174): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-04-15T15:13:32.985 INFO:teuthology.orchestra.run.smithi078.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 256 kB/s | 32 kB 00:00 2024-04-15T15:13:33.010 INFO:teuthology.orchestra.run.smithi078.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 119 kB/s | 15 kB 00:00 2024-04-15T15:13:33.044 INFO:teuthology.orchestra.run.smithi078.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-04-15T15:13:33.085 INFO:teuthology.orchestra.run.smithi078.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-15T15:13:33.111 INFO:teuthology.orchestra.run.smithi078.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-15T15:13:33.127 INFO:teuthology.orchestra.run.smithi078.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-15T15:13:33.211 INFO:teuthology.orchestra.run.smithi078.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-04-15T15:13:33.236 INFO:teuthology.orchestra.run.smithi078.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-15T15:13:33.295 INFO:teuthology.orchestra.run.smithi078.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-15T15:13:33.320 INFO:teuthology.orchestra.run.smithi078.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-15T15:13:33.379 INFO:teuthology.orchestra.run.smithi078.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-04-15T15:13:33.404 INFO:teuthology.orchestra.run.smithi078.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-15T15:13:33.463 INFO:teuthology.orchestra.run.smithi078.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-04-15T15:13:33.546 INFO:teuthology.orchestra.run.smithi078.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 556 kB/s | 46 kB 00:00 2024-04-15T15:13:33.581 INFO:teuthology.orchestra.run.smithi078.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-04-15T15:13:33.647 INFO:teuthology.orchestra.run.smithi078.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-04-15T15:13:33.664 INFO:teuthology.orchestra.run.smithi078.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 692 kB/s | 58 kB 00:00 2024-04-15T15:13:33.714 INFO:teuthology.orchestra.run.smithi078.stdout:(135/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-04-15T15:13:33.756 INFO:teuthology.orchestra.run.smithi078.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-15T15:13:33.773 INFO:teuthology.orchestra.run.smithi078.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-15T15:13:33.815 INFO:teuthology.orchestra.run.smithi078.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 894 kB/s | 90 kB 00:00 2024-04-15T15:13:33.840 INFO:teuthology.orchestra.run.smithi078.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-04-15T15:13:33.857 INFO:teuthology.orchestra.run.smithi078.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 366 kB/s | 31 kB 00:00 2024-04-15T15:13:33.917 INFO:teuthology.orchestra.run.smithi078.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-15T15:13:33.933 INFO:teuthology.orchestra.run.smithi078.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-04-15T15:13:33.949 INFO:teuthology.orchestra.run.smithi078.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-04-15T15:13:34.000 INFO:teuthology.orchestra.run.smithi078.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-04-15T15:13:34.033 INFO:teuthology.orchestra.run.smithi078.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-15T15:13:34.059 INFO:teuthology.orchestra.run.smithi078.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-04-15T15:13:34.100 INFO:teuthology.orchestra.run.smithi078.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-15T15:13:34.126 INFO:teuthology.orchestra.run.smithi078.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-04-15T15:13:34.151 INFO:teuthology.orchestra.run.smithi078.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-15T15:13:34.702 INFO:teuthology.orchestra.run.smithi078.stdout:(150/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 5.7 MB/s | 3.1 MB 00:00 2024-04-15T15:13:34.744 INFO:teuthology.orchestra.run.smithi078.stdout:(151/174): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-04-15T15:13:34.802 INFO:teuthology.orchestra.run.smithi078.stdout:(152/174): NetworkManager-1.47.4-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-04-15T15:13:34.844 INFO:teuthology.orchestra.run.smithi078.stdout:(153/174): NetworkManager-libnm-1.47.4-1.el9.x8 19 MB/s | 1.8 MB 00:00 2024-04-15T15:13:34.869 INFO:teuthology.orchestra.run.smithi078.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 633 kB/s | 41 kB 00:00 2024-04-15T15:13:34.894 INFO:teuthology.orchestra.run.smithi078.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-04-15T15:13:34.920 INFO:teuthology.orchestra.run.smithi078.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-15T15:13:34.970 INFO:teuthology.orchestra.run.smithi078.stdout:(157/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 4.0 MB/s | 3.4 MB 00:00 2024-04-15T15:13:34.995 INFO:teuthology.orchestra.run.smithi078.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 485 kB/s | 36 kB 00:00 2024-04-15T15:13:35.021 INFO:teuthology.orchestra.run.smithi078.stdout:(159/174): acl-2.3.1-4.el9.x86_64.rpm 570 kB/s | 71 kB 00:00 2024-04-15T15:13:35.063 INFO:teuthology.orchestra.run.smithi078.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 554 kB/s | 23 kB 00:00 2024-04-15T15:13:35.088 INFO:teuthology.orchestra.run.smithi078.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-04-15T15:13:35.113 INFO:teuthology.orchestra.run.smithi078.stdout:(162/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.3 MB/s | 192 kB 00:00 2024-04-15T15:13:35.138 INFO:teuthology.orchestra.run.smithi078.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-04-15T15:13:35.164 INFO:teuthology.orchestra.run.smithi078.stdout:(164/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 108 kB 00:00 2024-04-15T15:13:35.190 INFO:teuthology.orchestra.run.smithi078.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-04-15T15:13:35.214 INFO:teuthology.orchestra.run.smithi078.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 847 kB/s | 64 kB 00:00 2024-04-15T15:13:35.239 INFO:teuthology.orchestra.run.smithi078.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-04-15T15:13:35.315 INFO:teuthology.orchestra.run.smithi078.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2024-04-15T15:13:35.357 INFO:teuthology.orchestra.run.smithi078.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 15 MB/s | 2.2 MB 00:00 2024-04-15T15:13:35.394 INFO:teuthology.orchestra.run.smithi078.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-04-15T15:13:35.415 INFO:teuthology.orchestra.run.smithi078.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-04-15T15:13:35.466 INFO:teuthology.orchestra.run.smithi078.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-04-15T15:13:35.499 INFO:teuthology.orchestra.run.smithi078.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-04-15T15:13:35.541 INFO:teuthology.orchestra.run.smithi078.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 592 kB/s | 74 kB 00:00 2024-04-15T15:13:35.546 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:13:35.546 INFO:teuthology.orchestra.run.smithi078.stdout:Total 13 MB/s | 228 MB 00:17 2024-04-15T15:13:35.919 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2024-04-15T15:13:36.192 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2024-04-15T15:13:36.192 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2024-04-15T15:13:38.227 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2024-04-15T15:13:38.228 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2024-04-15T15:13:38.582 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-15T15:13:38.686 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:13:38.746 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:13:38.783 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:38.794 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:38.849 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:13:38.906 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:13:38.946 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:13:38.979 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:13:39.158 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:13:39.687 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:13:39.740 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:39.873 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:39.900 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:39.998 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:40.029 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:13:40.066 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:13:40.272 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:13:40.327 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:13:40.354 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:40.444 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:40.472 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:40.521 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:40.551 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:13:40.601 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:40.617 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:40.763 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:40.793 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:40.855 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:13:41.025 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:13:41.155 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:41.198 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:41.300 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:41.350 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:41.562 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:41.675 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:13:41.770 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:13:41.805 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:13:41.831 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:13:41.867 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:13:41.944 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:13:42.005 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:13:42.042 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:13:42.108 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:13:42.157 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:13:42.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:13:42.234 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:13:42.261 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:13:42.333 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:13:42.366 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:13:42.410 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:13:42.438 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:13:42.510 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:13:42.553 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:13:42.621 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:13:42.759 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:13:42.814 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:13:43.370 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:13:43.445 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:13:43.517 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:13:43.569 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:43.579 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:43.611 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:13:43.659 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:13:43.697 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:13:43.727 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:13:43.999 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:13:44.174 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:13:44.254 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:13:44.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:13:44.527 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:13:44.603 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:13:44.659 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:13:44.732 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:13:44.761 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:13:44.798 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:13:44.846 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:13:44.881 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:13:44.924 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:13:44.975 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:13:45.115 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:13:45.172 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:13:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2024-04-15T15:13:45.244 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:13:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:13:45.341 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:13:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:13:45.376 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:13:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:45.417 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:13:45.434 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:13:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:13:45.450 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:13:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:13:45.530 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:13:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:13:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:13:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:13:45.665 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:13:45.779 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:13:45.817 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:13:45.875 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:13:45.931 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:13:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:13:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:46.133 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:46.204 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:13:46.252 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:46.278 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:13:46.314 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:13:46.397 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:13:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:13:46.539 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:13:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:13:46.578 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:13:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:46.610 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:13:46.648 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:46.668 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:13:46.673 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:46.694 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:13:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:46.734 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:13:46.765 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:13:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:46.795 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:13:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:13:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:46.874 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:13:46.905 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:13:46.939 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:13:46.989 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:13:47.023 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:13:47.024 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:47.054 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:13:47.115 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:13:47.205 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:13:47.246 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:13:47.278 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:13:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:13:47.325 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:13:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:47.433 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:13:47.488 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:13:47.489 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:47.602 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:47.649 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:47.905 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:48.270 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:13:48.274 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:13:48.318 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:13:48.352 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:13:48.366 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:13:48.395 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:13:48.400 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:13:48.432 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:13:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:13:48.483 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:13:48.520 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:13:48.547 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:13:48.602 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:13:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:13:48.705 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:13:48.712 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:13:48.735 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:13:48.758 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:13:48.801 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:13:48.815 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:13:48.844 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:13:48.879 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:13:48.950 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:13:48.984 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:13:49.027 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:13:49.056 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:13:49.125 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:13:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:13:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:13:49.393 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:13:49.447 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:13:49.492 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:13:49.619 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:13:49.967 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:13:50.042 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:13:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:13:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:50.206 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:50.239 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:13:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:13:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:13:50.354 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:13:50.856 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:13:50.897 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:13:50.994 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:13:51.025 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:13:51.056 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:13:51.069 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:13:51.084 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:13:51.147 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:13:51.199 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:13:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:13:51.411 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:13:51.412 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:13:51.439 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:51.465 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:13:51.530 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:51.537 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:13:51.560 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:13:51.569 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:13:51.597 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:13:51.606 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:13:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:13:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:13:51.731 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:13:51.782 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:13:51.923 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:13:51.971 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:13:52.003 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:13:52.042 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:13:52.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:13:52.117 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:52.136 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:13:52.168 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:13:52.179 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:52.202 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:13:52.208 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:13:52.234 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:13:52.258 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:13:52.344 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:13:52.492 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:13:52.599 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:13:52.641 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:13:52.689 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:13:52.745 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:13:52.930 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:13:52.979 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:13:53.114 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:13:53.161 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:13:53.193 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:13:53.224 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:53.251 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:13:53.294 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:13:53.341 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:13:53.373 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:13:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:13:53.478 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:13:53.513 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:13:53.546 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:13:53.596 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:13:53.631 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:13:53.719 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:13:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:13:53.861 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:13:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:13:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:13:54.036 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:13:54.103 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:13:54.526 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:54.564 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:54.826 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:13:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:13:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:13:54.957 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:13:55.038 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:13:55.069 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:13:55.153 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:55.193 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:55.246 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:13:55.268 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:55.268 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:13:55.268 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:13:55.268 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:13:55.274 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:13:55.302 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:55.352 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:13:55.984 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:13:56.097 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:13:57.312 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:13:57.391 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:13:57.422 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:13:57.461 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:13:57.497 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:13:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:13:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:57.926 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:57.958 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:13:57.994 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:13:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:13:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:58.568 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:58.591 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:14:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:14:00.837 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:14:01.374 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:14:01.407 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:14:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:14:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:14:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:14:01.500 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:01.541 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-04-15T15:14:07.424 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:07.591 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:14:07.607 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:14:07.607 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:07.607 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:14:07.607 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:09.136 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:14:09.163 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:14:09.163 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:09.163 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:14:09.163 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:14:09.163 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:14:09.163 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:09.403 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:14:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:14:09.427 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:09.427 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:14:09.427 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:14:09.427 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:14:09.427 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:09.896 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:14:09.919 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:14:09.919 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:09.919 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:14:09.919 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:14:09.919 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:14:09.919 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:10.006 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:14:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:14:10.504 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:14:10.537 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:14:10.599 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:14:10.627 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:14:11.720 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:14:11.763 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:14:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:14:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:14:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:14:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:14:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:14:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:14:13.139 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:13.139 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:14:13.139 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:14:13.139 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:14:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:14:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:14:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:14:13.393 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /root 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2024-04-15T15:14:13.394 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:13.563 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:14:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:14:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:13.579 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:14:13.579 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:14.174 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:14:14.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:14:14.225 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:14:14.225 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:14.225 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:14:14.225 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:14:14.225 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:14:14.225 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:14.268 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:14:14.291 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:14:14.291 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:14.291 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:14:14.291 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:14.336 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:14.359 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:14.359 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:14.359 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:14:14.359 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:14:14.359 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:14:14.360 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:14.678 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:14:14.701 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:14:14.702 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:14.702 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:14:14.702 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:14:14.702 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:14:14.702 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:15.066 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:14:15.092 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:14:15.093 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:15.093 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:14:15.093 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:14:15.093 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:14:15.093 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:15.333 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:14:15.356 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:15.801 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:14:15.824 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:14:15.824 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:15.824 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:14:15.824 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:14:15.824 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:14:15.824 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:14:16.363 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:14:16.392 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:14:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:14:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:14:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:14:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:14:17.592 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:14:18.553 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:14:18.584 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:14:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:14:18.855 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:14:18.886 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:14:18.910 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:14:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:14:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:14:18.910 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:14:18.911 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:18.953 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:14:18.965 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:14:19.019 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:14:19.516 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:14:19.558 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:14:19.589 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:14:19.617 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:14:19.809 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:14:19.893 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:14:19.927 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:14:19.930 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:14:19.953 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:14:19.954 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:19.954 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:14:19.954 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:14:19.954 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:14:19.955 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:19.972 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:14:19.998 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:14:20.019 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:14:20.020 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:14:20.020 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:20.020 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:14:20.020 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:20.080 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:14:20.082 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:20.105 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:20.105 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:20.105 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:14:20.105 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:14:20.105 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:14:20.105 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:20.123 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:14:20.164 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:14:20.196 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:14:20.226 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:14:20.296 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:14:20.365 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:14:20.399 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:14:20.407 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:14:20.430 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:14:20.430 INFO:teuthology.orchestra.run.smithi078.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:14:20.430 INFO:teuthology.orchestra.run.smithi078.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:14:20.431 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:14:20.431 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:14:20.431 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:21.022 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:14:21.056 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:14:21.077 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:14:21.964 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:14:22.051 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:14:22.094 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:14:22.187 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:14:22.222 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:14:22.246 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:14:22.272 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:14:22.298 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:14:22.299 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:14:22.370 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:14:22.416 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:14:22.416 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:14:22.500 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:14:22.536 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:14:22.568 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:14:22.594 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:14:22.619 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:14:22.654 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:14:22.714 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:14:22.748 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:14:22.773 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:14:22.816 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:14:22.841 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:22.866 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:14:22.873 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:22.873 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:23.069 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:23.070 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:14:23.152 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:14:23.184 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:23.210 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:14:23.236 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:14:23.269 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:14:23.339 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:24.987 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:14:25.028 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:14:25.058 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:14:25.086 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:14:25.269 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:14:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:14:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:14:25.480 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:14:25.534 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:14:25.569 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:14:25.601 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:14:25.633 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:14:25.663 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:14:25.729 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:14:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:14:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:14:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:14:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:14:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:14:27.305 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:14:27.397 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:14:27.437 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:14:27.528 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:14:27.567 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:14:27.600 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:14:27.625 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:14:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:14:27.652 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:14:27.731 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:14:27.761 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:14:27.761 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:14:27.856 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:14:27.889 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:14:27.923 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:14:27.958 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:14:27.982 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:14:28.018 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:14:28.055 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:14:28.095 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:14:28.120 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:14:28.146 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:14:28.179 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:28.205 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:14:28.212 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:28.212 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:28.399 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:28.400 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:14:28.482 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:14:28.514 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:28.541 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:14:28.567 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:14:28.592 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:14:28.657 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:28.817 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:14:28.821 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:14:28.822 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:14:28.823 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:14:28.824 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:14:28.825 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:14:28.826 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:14:28.827 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:14:28.828 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:14:28.829 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:14:28.830 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:14:28.831 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:14:28.832 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:14:28.833 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:14:31.856 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:14:31.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:31.856 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-04-15T15:14:31.856 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.857 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:14:31.858 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.860 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.861 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.861 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:31.861 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:14:31.861 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:14:31.861 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:14:31.862 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:14:31.862 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:14:31.862 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:14:31.862 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:14:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:14:31.864 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:14:31.865 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:14:31.866 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:14:31.867 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:14:31.868 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:14:31.869 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:14:31.870 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:14:31.871 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:14:31.871 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:14:31.871 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:14:31.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:31.871 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-15T15:14:32.350 DEBUG:teuthology.parallel:result is None 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:14:34.206 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:14:34.207 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:14:34.208 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:14:34.209 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:14:34.210 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:14:34.211 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:14:34.212 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:14:34.213 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:14:34.214 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:14:34.215 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:14:34.216 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:14:34.217 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:14:34.218 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:14:34.219 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:14:34.220 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:14:34.221 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:14:37.204 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:14:37.204 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:37.204 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:14:37.205 INFO:teuthology.orchestra.run.smithi078.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.206 INFO:teuthology.orchestra.run.smithi078.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.207 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.208 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:14:37.209 INFO:teuthology.orchestra.run.smithi078.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:14:37.210 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:14:37.211 INFO:teuthology.orchestra.run.smithi078.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:14:37.212 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:14:37.213 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:14:37.214 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:14:37.215 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:14:37.216 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:14:37.217 INFO:teuthology.orchestra.run.smithi078.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:14:37.218 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:14:37.222 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:14:37.222 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:14:37.222 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:14:37.223 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:14:37.223 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:37.223 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2024-04-15T15:14:37.728 DEBUG:teuthology.parallel:result is None 2024-04-15T15:14:37.729 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:14:37.877 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:14:37.903 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:37.904 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:37.904 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:14:37.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:14:38.163 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:14:38.188 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:38.188 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:38.188 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:14:38.190 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-15T15:14:38.190 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:38.191 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:14:38.227 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:38.227 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:14:38.261 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-15T15:14:38.261 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:38.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:14:38.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:14:38.371 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:38.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:14:38.401 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:14:38.470 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-15T15:14:38.471 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:38.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:14:38.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:14:38.572 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:38.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:14:38.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:14:38.670 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-15T15:14:38.671 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:38.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:14:38.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:14:38.768 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:38.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:14:38.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:14:38.865 INFO:teuthology.run_tasks:Running task ceph... 2024-04-15T15:14:38.960 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-15T15:14:38.960 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-04-15T15:14:38.962 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-04-15T15:14:38.990 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-15T15:14:38.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:14:39.033 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:14:39.065 INFO:tasks.ceph:Creating extra log directories... 2024-04-15T15:14:39.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:14:39.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:14:39.140 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-15T15:14:39.141 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-15T15:14:39.141 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-15T15:14:39.141 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:39.141 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:14:39.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:39.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:39.336 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:39.336 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:14:39.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:39.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:39.508 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-15T15:14:39.508 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-15T15:14:39.508 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123', 'branch': 'wip-lusov-quiesce-xlock', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce}', 'email': None, 'first_in_suite': False, 'job_id': '7657123', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-quiesce-xlock'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-quiesce-xlock', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2288, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-lusov-quiesce-xlock', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9vR3gjx85IDEfOdCoscp48OpEZQiPGteXc/kLsFQoKsj1FtDW56a5VP1VFl1/MZbE3p1Oqz2pcwSST7cMdE0Y=', 'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWnxiGt5uOpN3O37WMgvVBLr5l137jxAV/h1H3Zhujy7KjZ40v1QMx6J6bXcq6NOBEL8EhbD9c3MroETUQ4HjQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_quiesce']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-15_14:46:31', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-15T15:14:39.509 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:14:39.511 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:14:39.557 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:14:39.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:14:39.625 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:39.625 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:14:39.647 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:14:39.648 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-04-15T15:14:39.702 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 805 Links: 1 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-15 15:14:23.680239211 +0000 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-15 15:12:23.370644667 +0000 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-15 15:12:23.370644667 +0000 2024-04-15T15:14:39.703 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-15 15:12:23.370644667 +0000 2024-04-15T15:14:39.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:14:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-15T15:14:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-15T15:14:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.7203e-05 s, 5.3 MB/s 2024-04-15T15:14:39.770 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:14:39.827 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 824 Links: 1 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-15 15:14:23.681239200 +0000 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-15 15:12:23.730640447 +0000 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-15 15:12:23.730640447 +0000 2024-04-15T15:14:39.882 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-15 15:12:23.730640447 +0000 2024-04-15T15:14:39.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:14:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-15T15:14:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-15T15:14:39.951 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.0173e-05 s, 5.7 MB/s 2024-04-15T15:14:39.952 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:14:40.007 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 844 Links: 1 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-15 15:14:23.681239200 +0000 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-15 15:12:24.074636415 +0000 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-15 15:12:24.074636415 +0000 2024-04-15T15:14:40.063 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-15 15:12:24.074636415 +0000 2024-04-15T15:14:40.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:14:40.128 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-15T15:14:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-15T15:14:40.129 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.4862e-05 s, 5.4 MB/s 2024-04-15T15:14:40.130 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:14:40.186 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-04-15T15:14:40.241 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 863 Links: 1 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-15 15:14:23.681239200 +0000 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-15 15:12:24.412632453 +0000 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-15 15:12:24.412632453 +0000 2024-04-15T15:14:40.242 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-15 15:12:24.412632453 +0000 2024-04-15T15:14:40.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:14:40.308 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-15T15:14:40.308 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-15T15:14:40.308 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.4395e-05 s, 5.4 MB/s 2024-04-15T15:14:40.309 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:14:40.365 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:40.365 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:40.365 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:14:40.383 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:14:40.384 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-04-15T15:14:40.440 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:14:40.440 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:40.440 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-15T15:14:40.440 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:40.440 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:40.441 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-04-15 15:14:29.011488796 +0000 2024-04-15T15:14:40.441 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-04-15 15:12:12.062412355 +0000 2024-04-15T15:14:40.441 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-04-15 15:12:12.062412355 +0000 2024-04-15T15:14:40.441 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-04-15 15:12:12.062412355 +0000 2024-04-15T15:14:40.441 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:14:40.509 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-04-15T15:14:40.509 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-04-15T15:14:40.509 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00020856 s, 2.5 MB/s 2024-04-15T15:14:40.510 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:14:40.568 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-04-15T15:14:40.624 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-04-15 15:14:29.012488780 +0000 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-04-15 15:12:12.412406368 +0000 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-04-15 15:12:12.412406368 +0000 2024-04-15T15:14:40.625 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-04-15 15:12:12.412406368 +0000 2024-04-15T15:14:40.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:14:40.691 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-04-15T15:14:40.692 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-04-15T15:14:40.692 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00024686 s, 2.1 MB/s 2024-04-15T15:14:40.693 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:14:40.750 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-04-15 15:14:29.012488780 +0000 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-04-15 15:12:12.739400775 +0000 2024-04-15T15:14:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-04-15 15:12:12.739400775 +0000 2024-04-15T15:14:40.807 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-04-15 15:12:12.739400775 +0000 2024-04-15T15:14:40.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:14:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-04-15T15:14:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-04-15T15:14:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000206053 s, 2.5 MB/s 2024-04-15T15:14:40.874 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:14:40.931 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-04-15 15:14:29.012488780 +0000 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-04-15 15:12:13.071395096 +0000 2024-04-15T15:14:40.986 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-04-15 15:12:13.071395096 +0000 2024-04-15T15:14:40.987 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2024-04-15 15:12:13.071395096 +0000 2024-04-15T15:14:40.987 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:14:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-04-15T15:14:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-04-15T15:14:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000162221 s, 3.2 MB/s 2024-04-15T15:14:41.054 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:14:41.110 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:41.111 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:14:41.111 INFO:tasks.ceph:Generating config... 2024-04-15T15:14:41.112 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-15T15:14:41.112 INFO:tasks.ceph:[client] debug client = 20 2024-04-15T15:14:41.112 INFO:tasks.ceph:[client] debug ms = 1 2024-04-15T15:14:41.112 INFO:tasks.ceph:[client] pid file = 2024-04-15T15:14:41.112 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-15T15:14:41.112 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-15T15:14:41.112 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-15T15:14:41.113 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-15T15:14:41.114 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-15T15:14:41.114 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-15T15:14:41.114 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-15T15:14:41.114 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-15T15:14:41.115 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-15T15:14:41.115 INFO:tasks.ceph:Setting up mon.a... 2024-04-15T15:14:41.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:41.171 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2024-04-15T15:14:41.174 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-15T15:14:41.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:14:41.251 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46'), ('mon.b', '172.21.15.78'), ('mon.c', '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]')] 2024-04-15T15:14:41.251 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46,172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-15T15:14:41.252 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:41.252 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:14:41.308 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --add b 172.21.15.78 --addv c '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:41.393 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 95f1e89e-6559-430a-8622-bd018dcc0afd 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 95f1e89e-6559-430a-8622-bd018dcc0afd 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2024-04-15T15:14:41.393205+0000 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-15T15:14:41.393205+0000 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 17 (quincy) 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:1: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.b 2024-04-15T15:14:41.394 INFO:teuthology.orchestra.run.smithi046.stdout:2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-04-15T15:14:41.395 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-15T15:14:41.396 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:14:41.412 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 95f1e89e-6559-430a-8622-bd018dcc0afd... 2024-04-15T15:14:41.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:14:41.455 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:14:41.505 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-04-15T15:14:41.505 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-04-15T15:14:41.505 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:14:41.505 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2024-04-15T15:14:41.505 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.505 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-04-15T15:14:41.506 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:14:41.507 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790]" 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 95f1e89e-6559-430a-8622-bd018dcc0afd 2024-04-15T15:14:41.508 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-04-15T15:14:41.509 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-04-15T15:14:41.510 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:14:41.511 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-04-15T15:14:41.513 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:14:41.513 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:14:41.513 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-04-15T15:14:41.513 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-04-15T15:14:41.513 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-04-15T15:14:41.514 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:41.515 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: mon down mkfs grace = 300 2024-04-15T15:14:41.516 INFO:teuthology.orchestra.run.smithi046.stdout: mon op complaint time = 120 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: client mount timeout = 600 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-04-15T15:14:41.517 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = "" 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout:[mds] 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 20 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 20 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-04-15T15:14:41.518 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug frag = True 2024-04-15T15:14:41.519 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug scatterstat = True 2024-04-15T15:14:41.519 INFO:teuthology.orchestra.run.smithi046.stdout: mds op complaint time = 180 2024-04-15T15:14:41.519 INFO:teuthology.orchestra.run.smithi046.stdout: mds verify scatter = True 2024-04-15T15:14:41.519 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2024-04-15T15:14:41.519 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2024-04-15T15:14:41.519 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.520 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.521 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:14:41.522 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "172.21.15.46,172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790]" 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 95f1e89e-6559-430a-8622-bd018dcc0afd 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.523 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.524 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 20 2024-04-15T15:14:41.525 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 20 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 10 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: osd op complaint time = 180 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-04-15T15:14:41.527 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug client = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-04-15T15:14:41.528 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.529 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: mon down mkfs grace = 300 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: mon op complaint time = 120 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: client mount timeout = 600 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: debug client = 20 2024-04-15T15:14:41.530 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = "" 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: rados mon op timeout = 900 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: rados osd op timeout = 900 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout:[mds] 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: debug mds = 20 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: debug mds balancer = 20 2024-04-15T15:14:41.531 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: mds debug frag = True 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: mds debug scatterstat = True 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: mds op complaint time = 180 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: mds verify scatter = True 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: osd op complaint time = 180 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: rados mon op timeout = 900 2024-04-15T15:14:41.532 INFO:teuthology.orchestra.run.smithi078.stdout: rados osd op timeout = 900 2024-04-15T15:14:41.533 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-15T15:14:41.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-15T15:14:41.631 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-15T15:14:41.632 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:41.632 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-15T15:14:41.647 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:41.647 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-15T15:14:41.702 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:14:41.703 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:41.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:14:41.703 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:14:41.785 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:14:41.785 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:41.841 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:14:41.841 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:41.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:14:41.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:14:41.883 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:14:41.883 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:41.938 INFO:tasks.ceph:Setting up mon nodes... 2024-04-15T15:14:41.939 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-15T15:14:41.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:14:41.995 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:14:41.998 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:14:42.065 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:14:42.067 INFO:tasks.ceph:Setting up mds nodes... 2024-04-15T15:14:42.067 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:14:42.124 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:14:42.127 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-15T15:14:42.155 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:14:42.252 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:14:42.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-15T15:14:42.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:14:42.340 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:14:42.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-15T15:14:42.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:14:42.468 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:14:42.470 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-15T15:14:42.498 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-15T15:14:42.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T15:14:42.543 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-15T15:14:42.559 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-15T15:14:42.643 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-15T15:14:42.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T15:14:42.702 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-15T15:14:42.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-15T15:14:42.762 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-15T15:14:42.778 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-15T15:14:42.778 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:14:42.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:42.806 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:42.806 INFO:tasks.ceph:role: osd.0 2024-04-15T15:14:42.806 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:14:42.806 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:14:42.874 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:42.874 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:42.874 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:42.875 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:42.908 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-04-15T15:14:42.909 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-04-15T15:14:42.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:42.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:43.059 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:43.129 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:43.129 INFO:tasks.ceph:role: osd.1 2024-04-15T15:14:43.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:14:43.129 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:43.197 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:43.229 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-04-15T15:14:43.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-04-15T15:14:43.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:43.308 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:43.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:43.449 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:43.449 INFO:tasks.ceph:role: osd.2 2024-04-15T15:14:43.450 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:14:43.450 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:43.517 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:43.518 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:43.518 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:43.550 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-04-15T15:14:43.554 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-04-15T15:14:43.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:43.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:43.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:43.766 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:43.767 INFO:tasks.ceph:role: osd.3 2024-04-15T15:14:43.767 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:14:43.767 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:43.834 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:43.866 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-04-15T15:14:43.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-04-15T15:14:43.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:43.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:44.012 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:44.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:44.108+0000 7fa78e8b3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-15T15:14:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:44.108+0000 7fa78e8b3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-15T15:14:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:44.108+0000 7fa78e8b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:14:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:44.108+0000 7fa78e8b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:14:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:44.108+0000 7fa78e8b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:14:44.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:44.108+0000 7fa78e8b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-15T15:14:46.960 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:46.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:47.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:47.086+0000 7f884ac19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-15T15:14:47.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:47.086+0000 7f884ac19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-15T15:14:47.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:47.086+0000 7f884ac19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:47.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:47.086+0000 7f884ac19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:47.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:47.086+0000 7f884ac19640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:47.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:47.086+0000 7f884ac19640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-15T15:14:49.932 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:49.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:50.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:50.057+0000 7f59036aa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-15T15:14:50.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:50.057+0000 7f59036aa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-15T15:14:50.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:50.057+0000 7f59036aa640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:50.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:50.057+0000 7f59036aa640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:50.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:50.057+0000 7f59036aa640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:50.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:50.057+0000 7f59036aa640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-15T15:14:52.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:52.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:53.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:53.032+0000 7f6cd281b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-15T15:14:53.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:53.032+0000 7f6cd281b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-15T15:14:53.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:53.032+0000 7f6cd281b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:53.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:53.032+0000 7f6cd281b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:53.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:53.032+0000 7f6cd281b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:53.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:14:53.033+0000 7f6cd281b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-15T15:14:55.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:55.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:55.944 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:55.944 INFO:tasks.ceph:role: osd.4 2024-04-15T15:14:55.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:14:55.945 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:56.530 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-04-15T15:14:56.532 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-04-15T15:14:56.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:56.650 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:56.686 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:56.757 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:56.758 INFO:tasks.ceph:role: osd.5 2024-04-15T15:14:56.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:14:56.758 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:14:56.833 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:56.833 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:57.294 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-04-15T15:14:57.298 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-04-15T15:14:57.298 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:57.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:57.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:57.491 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:57.491 INFO:tasks.ceph:role: osd.6 2024-04-15T15:14:57.491 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:14:57.491 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:57.567 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:57.568 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:58.063 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-04-15T15:14:58.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-04-15T15:14:58.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:58.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:58.236 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:58.305 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:58.305 INFO:tasks.ceph:role: osd.7 2024-04-15T15:14:58.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:14:58.305 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:14:58.380 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:58.381 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:58.913 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-04-15T15:14:58.915 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-04-15T15:14:58.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:59.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:59.086 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:14:59.186+0000 7fd47460f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-15T15:14:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:14:59.186+0000 7fd47460f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-15T15:14:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:14:59.186+0000 7fd47460f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:14:59.186+0000 7fd47460f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:14:59.186+0000 7fd47460f640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:14:59.186+0000 7fd47460f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-15T15:15:02.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-15T15:15:02.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:15:02.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:02.168+0000 7f3487738640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-15T15:15:02.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:02.169+0000 7f3487738640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-15T15:15:02.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:02.169+0000 7f3487738640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:15:02.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:02.169+0000 7f3487738640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:15:02.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:02.169+0000 7f3487738640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:15:02.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:02.169+0000 7f3487738640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-15T15:15:05.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-15T15:15:05.048 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:15:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:05.139+0000 7f7217414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-15T15:15:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:05.140+0000 7f7217414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-15T15:15:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:05.140+0000 7f7217414640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:15:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:05.140+0000 7f7217414640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:15:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:05.140+0000 7f7217414640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:15:05.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:05.140+0000 7f7217414640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-15T15:15:07.992 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-15T15:15:08.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:15:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:08.111+0000 7fb904526640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-15T15:15:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:08.112+0000 7fb904526640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-15T15:15:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:08.112+0000 7fb904526640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:15:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:08.112+0000 7fb904526640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:15:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:08.112+0000 7fb904526640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:15:08.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:08.112+0000 7fb904526640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-15T15:15:09.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:09.513 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:10.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-15T15:15:11.004 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-15T15:15:11.005 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.005 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-15T15:15:11.039 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.039 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-15T15:15:11.112 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.112 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-15T15:15:11.183 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-15T15:15:11.253 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-15T15:15:11.323 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.324 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-15T15:15:11.394 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-15T15:15:11.462 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.462 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-15T15:15:11.495 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-15T15:15:11.566 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-15T15:15:11.638 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-15T15:15:11.709 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-15T15:15:11.778 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-15T15:15:11.848 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-15T15:15:11.917 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.917 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-15T15:15:11.935 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:11.936 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-15T15:15:11.993 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:11.994 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-15T15:15:12.010 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:12.010 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-15T15:15:12.065 INFO:tasks.ceph:Adding keys to all mons... 2024-04-15T15:15:12.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:15:12.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:15:12.109 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.y] 2024-04-15T15:15:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBhRB1mABxSOxAAoEB3jK+dAm0BhB+SFLUyxA== 2024-04-15T15:15:12.109 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2024-04-15T15:15:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mf25mBxAARBEukB5oxK15VCtiDQswRA== 2024-04-15T15:15:12.109 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2024-04-15T15:15:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mw6IJDxAAnMgkvYN8tpc8lzkKtvdHfQ== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBkRB1m+syUBhAA7JRJ68ztnjHCymDvmiEn/A== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBnRB1mb0s/BRAA0klngWpZ4czyMeWil60J9Q== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBqRB1mRCF1AxAA5jX9o6usFucYJoqv0lCYOg== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBtRB1mb/P8ARAAyWXAjjMCwEdGc31rhPSsQg== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mw7fVAxAAugywxKnWYMnJYE7dZdjnsw== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1matdFFBAAygX2/zcEaHpiYvVmgO7aAg== 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2024-04-15T15:15:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1m8wrkGxAAQsEA5qc7LD6mnJwLrlZ79Q== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBzRB1mZQA1CxAAlLXltfdCGtIXiPjxx9QNWA== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB2RB1mzBcsChAALZDAAAE8bfcxd6ZCqrbTZQ== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.6] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB5RB1mHYBlCBAAQ0q/qP2uBkHQ64gpR1n5vg== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.7] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQB8RB1mPvq3BhAALSJbv9Hdh+k0E/ZM1f9QSQ== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mChVZIBAAvN0/KdZG4Oam2AdTqRjJtw== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[client.2] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1m9TZLJhAAV1mJUfjsEt4HMgYroyY/cg== 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout:[client.1] 2024-04-15T15:15:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mkD/bKRAAmlMhj1H+/gj3v/l9r8wKkQ== 2024-04-15T15:15:12.112 INFO:teuthology.orchestra.run.smithi046.stdout:[client.3] 2024-04-15T15:15:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mDFJqLRAAkSUK13PRVTZMg2rT0wIXEg== 2024-04-15T15:15:12.132 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.y] 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBhRB1mABxSOxAAoEB3jK+dAm0BhB+SFLUyxA== 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a] 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1mf25mBxAARBEukB5oxK15VCtiDQswRA== 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c] 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1mw6IJDxAAnMgkvYN8tpc8lzkKtvdHfQ== 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBkRB1m+syUBhAA7JRJ68ztnjHCymDvmiEn/A== 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-04-15T15:15:12.133 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBnRB1mb0s/BRAA0klngWpZ4czyMeWil60J9Q== 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBqRB1mRCF1AxAA5jX9o6usFucYJoqv0lCYOg== 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBtRB1mb/P8ARAAyWXAjjMCwEdGc31rhPSsQg== 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1mw7fVAxAAugywxKnWYMnJYE7dZdjnsw== 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b] 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1matdFFBAAygX2/zcEaHpiYvVmgO7aAg== 2024-04-15T15:15:12.134 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1m8wrkGxAAQsEA5qc7LD6mnJwLrlZ79Q== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBzRB1mZQA1CxAAlLXltfdCGtIXiPjxx9QNWA== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQB2RB1mzBcsChAALZDAAAE8bfcxd6ZCqrbTZQ== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.6] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQB5RB1mHYBlCBAAQ0q/qP2uBkHQ64gpR1n5vg== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.7] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQB8RB1mPvq3BhAALSJbv9Hdh+k0E/ZM1f9QSQ== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1mChVZIBAAvN0/KdZG4Oam2AdTqRjJtw== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[client.2] 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1m9TZLJhAAV1mJUfjsEt4HMgYroyY/cg== 2024-04-15T15:15:12.135 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2024-04-15T15:15:12.136 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1mkD/bKRAAmlMhj1H+/gj3v/l9r8wKkQ== 2024-04-15T15:15:12.136 INFO:teuthology.orchestra.run.smithi078.stdout:[client.3] 2024-04-15T15:15:12.136 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBiRB1mDFJqLRAAkSUK13PRVTZMg2rT0wIXEg== 2024-04-15T15:15:12.137 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:15:12.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:15:12.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.294 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.351 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.354 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.524 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:15:12.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:15:12.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:15:12.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.808 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.882 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.939 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:15:12.995 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:12.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.114 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:15:13.232 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-15T15:15:13.232 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-15T15:15:13.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:15:13.360 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-15T15:15:13.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-15T15:15:13.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:15:13.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-15T15:15:13.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-15T15:15:13.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:15:13.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-15T15:15:13.741 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:15:13.743 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:15:13.796 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-15T15:15:13.797 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-15T15:15:13.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-15T15:15:13.799 INFO:tasks.ceph.mon.a:Started 2024-04-15T15:15:13.799 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-15T15:15:13.799 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-15T15:15:13.839 INFO:tasks.ceph.mon.b:Started 2024-04-15T15:15:13.839 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-15T15:15:13.839 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-15T15:15:13.841 INFO:tasks.ceph.mon.c:Started 2024-04-15T15:15:13.841 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-15T15:15:13.841 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-15T15:15:13.842 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-15T15:15:13.844 INFO:tasks.ceph.mgr.y:Started 2024-04-15T15:15:13.844 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-15T15:15:13.844 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-15T15:15:13.847 INFO:tasks.ceph.mgr.x:Started 2024-04-15T15:15:13.847 INFO:tasks.ceph:Setting crush tunables to default 2024-04-15T15:15:13.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2024-04-15T15:15:13.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.949+0000 7f3ca43cd640 1 Processor -- start 2024-04-15T15:15:13.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.949+0000 7f3ca43cd640 1 -- start start 2024-04-15T15:15:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca43cd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 0x7f3c9c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca43cd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c9c0b47a0 con 0x7f3c9c15eae0 2024-04-15T15:15:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca43cd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c9c0b4a80 con 0x7f3c9c15e700 2024-04-15T15:15:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca43cd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3c9c0b53b0 con 0x7f3c9c0d2360 2024-04-15T15:15:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca296a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3c9c15eae0 0x7f3c9c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60388/0 (socket says 172.21.15.46:60388) 2024-04-15T15:15:13.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca296a640 1 -- 172.21.15.46:0/2459386482 learned_addr learned my addr 172.21.15.46:0/2459386482 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:13.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca1968640 1 --2- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 0x7f3c9c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca1968640 1 -- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 msgr2=0x7f3c9c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:15:13.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca1968640 1 -- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 msgr2=0x7f3c9c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:15:13.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca1968640 1 --2- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 0x7f3c9c0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:15:13.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:13.950+0000 7f3ca1968640 1 --2- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 0x7f3c9c0d2b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.104+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 553659545 0 0) 0x7f3c9c0b4a80 con 0x7f3c9c15e700 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.104+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c88003520 con 0x7f3c9c15e700 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 <== mon.1 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2096459804 0 0) 0x7f3c88003520 con 0x7f3c9c15e700 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 msgr2=0x7f3c9c0d2b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 --2- 172.21.15.46:0/2459386482 >> v2:172.21.15.78:3300/0 conn(0x7f3c9c0d2360 0x7f3c9c0d2b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 >> v1:172.21.15.46:6789/0 conn(0x7f3c9c15eae0 legacy=0x7f3c9c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c0b4d60 con 0x7f3c9c15e700 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 <== mon.1 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f3c980016c0 con 0x7f3c9c15e700 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 >> v1:172.21.15.78:6789/0 conn(0x7f3c9c15e700 legacy=0x7f3c9c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003a00 0x7f3c88003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88004800 0x7f3c88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c88005230 0x7f3c8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c0b53b0 con 0x7f3c88003a00 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8800c190 con 0x7f3c88005230 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8800c390 con 0x7f3c88004800 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca2169640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003a00 0x7f3c88003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca296a640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c88005230 0x7f3c8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca1968640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88004800 0x7f3c88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca2169640 1 -- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88004800 msgr2=0x7f3c88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca2169640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88004800 0x7f3c88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca2169640 1 -- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c88005230 msgr2=0x7f3c8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca2169640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c88005230 0x7f3c8800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca2169640 1 -- 172.21.15.46:0/2459386482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c88003520 con 0x7f3c88003a00 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.105+0000 7f3ca296a640 1 -- 172.21.15.46:0/2459386482 reap_dead start 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.106+0000 7f3ca2169640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003a00 0x7f3c88003e40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c98007f20 tx=0x7f3c88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b752cd365d0940c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98039070 con 0x7f3c88003a00 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9802c940 con 0x7f3c88003a00 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca1167640 1 -- 172.21.15.46:0/2459386482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98007b40 con 0x7f3c88003a00 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca43cd640 1 -- 172.21.15.46:0/2459386482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003a00 msgr2=0x7f3c88003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca43cd640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003a00 0x7f3c88003e40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c98007f20 tx=0x7f3c88005830 comp rx=0 tx=0).stop 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca43cd640 1 -- 172.21.15.46:0/2459386482 shutdown_connections 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca43cd640 1 --2- 172.21.15.46:0/2459386482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c88003a00 0x7f3c88003e40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca43cd640 1 -- 172.21.15.46:0/2459386482 >> 172.21.15.46:0/2459386482 conn(0x7f3c9c08bf60 msgr2=0x7f3c9c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.107+0000 7f3ca43cd640 1 -- 172.21.15.46:0/2459386482 shutdown_connections 2024-04-15T15:15:14.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.108+0000 7f3ca43cd640 1 -- 172.21.15.46:0/2459386482 wait complete. 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.109+0000 7f3ca43cd640 1 Processor -- start 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.109+0000 7f3ca43cd640 1 -- start start 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88003a00 0x7f3c9c0cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c9c1f4260 0x7f3c9c161840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 0x7f3c9c15f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c0d1d40 con 0x7f3c9c154010 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9c0b4450 con 0x7f3c9c1f4260 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3c9c0d1ec0 con 0x7f3c88003a00 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 0x7f3c9c15f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 0x7f3c9c15f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37180/0 (socket says 172.21.15.46:37180) 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 -- 172.21.15.46:0/31359678 learned_addr learned my addr 172.21.15.46:0/31359678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca1968640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c9c1f4260 0x7f3c9c161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca2169640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88003a00 0x7f3c9c0cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 -- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88003a00 msgr2=0x7f3c9c0cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88003a00 0x7f3c9c0cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 -- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c9c1f4260 msgr2=0x7f3c9c161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c9c1f4260 0x7f3c9c161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 -- 172.21.15.46:0/31359678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c9c0ca860 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca296a640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 0x7f3c9c15f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c15efa0 tx=0x7f3c94007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400f9d0 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c94018070 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9c0cab40 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.110+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9400fdb0 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.111+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c9c161d80 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.111+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3c9401d460 con 0x7f3c9c154010 2024-04-15T15:15:14.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.111+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3c940137e0 con 0x7f3c9c154010 2024-04-15T15:15:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.111+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c9c0d5330 con 0x7f3c9c154010 2024-04-15T15:15:14.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.114+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3c9c0d5330 con 0x7f3c9c154010 2024-04-15T15:15:14.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.182+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3c9c06e8b0 con 0x7f3c9c154010 2024-04-15T15:15:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.183+0000 7f3c8effd640 1 -- 172.21.15.46:0/31359678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f3c94013a30 con 0x7f3c9c154010 2024-04-15T15:15:14.185 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-04-15T15:15:14.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 msgr2=0x7f3c9c15f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 0x7f3c9c15f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c15efa0 tx=0x7f3c94007a90 comp rx=0 tx=0).stop 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 shutdown_connections 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c9c154010 0x7f3c9c15f9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c9c1f4260 0x7f3c9c161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 --2- 172.21.15.46:0/31359678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c88003a00 0x7f3c9c0cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 >> 172.21.15.46:0/31359678 conn(0x7f3c9c08bf60 msgr2=0x7f3c9c0cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 shutdown_connections 2024-04-15T15:15:14.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:14.184+0000 7f3ca43cd640 1 -- 172.21.15.46:0/31359678 wait complete. 2024-04-15T15:15:14.197 INFO:tasks.ceph:check_enable_crimson: False 2024-04-15T15:15:14.197 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-15T15:15:14.197 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:14.197 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-15T15:15:14.225 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:14.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-15T15:15:14.294 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:14.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-15T15:15:14.364 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:14.364 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-15T15:15:14.432 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:14.433 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-15T15:15:14.461 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:14.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-15T15:15:14.529 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:14.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-15T15:15:14.586 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:14.585+0000 7f4a1243e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:15:14.597 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:15:14.598 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-15T15:15:14.618 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:14.616+0000 7fda0e23b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:15:14.642 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:14.640+0000 7f4a1243e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:15:14.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new c9d790af-808a-4ab7-8700-a7ccec98f3d9 0 2024-04-15T15:15:14.676 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:14.672+0000 7fda0e23b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:15:14.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.809+0000 7f13f78b8640 1 Processor -- start 2024-04-15T15:15:14.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.809+0000 7f13f78b8640 1 -- start start 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f78b8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 0x7f13f00b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f78b8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f00d1a10 0x7f13f00b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f78b8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f015ba70 con 0x7f13f00d08a0 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f78b8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f13f00ce5b0 con 0x7f13f015b690 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f78b8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f13f00ce890 con 0x7f13f00d1a10 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f5654640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 0x7f13f00b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f5e55640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f00d1a10 0x7f13f00b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f5654640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 0x7f13f00b6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40364/0 (socket says 172.21.15.78:40364) 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.810+0000 7f13f5654640 1 -- 172.21.15.78:0/2967445290 learned_addr learned my addr 172.21.15.78:0/2967445290 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.811+0000 7f13f5e55640 1 -- 172.21.15.78:0/2967445290 >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 msgr2=0x7f13f00b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.811+0000 7f13f5e55640 1 --2- 172.21.15.78:0/2967445290 >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 0x7f13f00b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.811+0000 7f13f5e55640 1 -- 172.21.15.78:0/2967445290 >> v1:172.21.15.46:6789/0 conn(0x7f13f00d08a0 legacy=0x7f13f00d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.811+0000 7f13f5e55640 1 -- 172.21.15.78:0/2967445290 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f00ceb70 con 0x7f13f00d1a10 2024-04-15T15:15:14.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.811+0000 7f13f5654640 1 --2- 172.21.15.78:0/2967445290 >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 0x7f13f00b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:14.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.811+0000 7f13f5e55640 1 --2- 172.21.15.78:0/2967445290 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f00d1a10 0x7f13f00b1660 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009dc0 tx=0x7f13e802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a020e4ef9eab6d2c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13e7fff640 1 -- 172.21.15.78:0/2967445290 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8039070 con 0x7f13f00d1a10 2024-04-15T15:15:14.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13e7fff640 1 -- 172.21.15.78:0/2967445290 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e802cc40 con 0x7f13f00d1a10 2024-04-15T15:15:14.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13e7fff640 1 -- 172.21.15.78:0/2967445290 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8034440 con 0x7f13f00d1a10 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 -- 172.21.15.78:0/2967445290 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f00d1a10 msgr2=0x7f13f00b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 --2- 172.21.15.78:0/2967445290 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f00d1a10 0x7f13f00b1660 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009dc0 tx=0x7f13e802be90 comp rx=0 tx=0).stop 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 -- 172.21.15.78:0/2967445290 shutdown_connections 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 --2- 172.21.15.78:0/2967445290 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f00d1a10 0x7f13f00b1660 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 --2- 172.21.15.78:0/2967445290 >> v2:172.21.15.78:3300/0 conn(0x7f13f015b690 0x7f13f00b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 -- 172.21.15.78:0/2967445290 >> 172.21.15.78:0/2967445290 conn(0x7f13f009a9d0 msgr2=0x7f13f00b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 -- 172.21.15.78:0/2967445290 shutdown_connections 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.812+0000 7f13f78b8640 1 -- 172.21.15.78:0/2967445290 wait complete. 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.813+0000 7f13f78b8640 1 Processor -- start 2024-04-15T15:15:14.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.813+0000 7f13f78b8640 1 -- start start 2024-04-15T15:15:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f78b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f00d08a0 0x7f13f0142d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f78b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f00d1a10 0x7f13f00b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f78b8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 0x7f13f01559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f78b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13f00b1ba0 con 0x7f13f00d1a10 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f78b8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f13f00d1010 con 0x7f13f00d08a0 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f78b8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f13f00b1d20 con 0x7f13f015b690 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f5e55640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 0x7f13f01559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f5654640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f00d08a0 0x7f13f0142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f5e55640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 0x7f13f01559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53844/0 (socket says 172.21.15.78:53844) 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f5e55640 1 -- 172.21.15.78:0/1864250529 learned_addr learned my addr 172.21.15.78:0/1864250529 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f4e53640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f00d1a10 0x7f13f00b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f5e55640 1 -- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f00d08a0 msgr2=0x7f13f0142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.814+0000 7f13f5e55640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f00d08a0 0x7f13f0142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f5e55640 1 -- 172.21.15.78:0/1864250529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f00d1a10 msgr2=0x7f13f00b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f5e55640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f00d1a10 0x7f13f00b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f5e55640 1 -- 172.21.15.78:0/1864250529 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f00b5580 con 0x7f13f015b690 2024-04-15T15:15:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f5654640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f00d08a0 0x7f13f0142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f5e55640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 0x7f13f01559a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009dc0 tx=0x7f13e8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8039050 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e8045070 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f00b5860 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8009450 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f0143270 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f13e8040690 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.815+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f13e804a8c0 con 0x7f13f015b690 2024-04-15T15:15:14.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.816+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f00690f0 con 0x7f13f015b690 2024-04-15T15:15:14.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.818+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f13e804a460 con 0x7f13f015b690 2024-04-15T15:15:14.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.890+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c9d790af-808a-4ab7-8700-a7ccec98f3d9", "id": 0} v 0) -- 0x7f13f0075330 con 0x7f13f015b690 2024-04-15T15:15:14.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.893+0000 7f13e5ffb640 1 -- 172.21.15.78:0/1864250529 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c9d790af-808a-4ab7-8700-a7ccec98f3d9", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f13e804ab10 con 0x7f13f015b690 2024-04-15T15:15:14.895 INFO:teuthology.orchestra.run.smithi078.stdout:0 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 msgr2=0x7f13f01559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 0x7f13f01559a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009dc0 tx=0x7f13e8004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 shutdown_connections 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13f015b690 0x7f13f01559a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13f00d1a10 0x7f13f00b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 --2- 172.21.15.78:0/1864250529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13f00d08a0 0x7f13f0142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 >> 172.21.15.78:0/1864250529 conn(0x7f13f009a9d0 msgr2=0x7f13f006f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 shutdown_connections 2024-04-15T15:15:14.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:14.894+0000 7f13f78b8640 1 -- 172.21.15.78:0/1864250529 wait complete. 2024-04-15T15:15:14.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new e6b51350-f913-4f16-ab9f-dbb94a6279ed 1 2024-04-15T15:15:15.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.007+0000 7f587cd50640 1 Processor -- start 2024-04-15T15:15:15.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.007+0000 7f587cd50640 1 -- start start 2024-04-15T15:15:15.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587cd50640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f58780b0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587cd50640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f58780aec60 0x7f58780af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587cd50640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58780afc70 con 0x7f587815b560 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587cd50640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f58780b0a40 con 0x7f58780aec60 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587cd50640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f58780af990 con 0x7f58780b00a0 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f586dd9b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f58780b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f5876d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f58780aec60 0x7f58780af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f587815b560 0x7f58780b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40316/0 (socket says 172.21.15.78:40316) 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f587659c640 1 -- 172.21.15.78:0/655653109 learned_addr learned my addr 172.21.15.78:0/655653109 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f5876d9d640 1 -- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 msgr2=0x7f58780b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f5876d9d640 1 --2- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f58780b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f5876d9d640 1 -- 172.21.15.78:0/655653109 >> v1:172.21.15.78:6789/0 conn(0x7f587815b560 legacy=0x7f58780b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f5875d9b640 1 -- 172.21.15.78:0/655653109 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645411434 0 0) 0x7f58780afc70 con 0x7f587815b560 2024-04-15T15:15:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.008+0000 7f5876d9d640 1 -- 172.21.15.78:0/655653109 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58780b42a0 con 0x7f58780aec60 2024-04-15T15:15:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f5876d9d640 1 --2- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58780aec60 0x7f58780af0f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5870009d90 tx=0x7f587002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8ce70450f228db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f5875d9b640 1 -- 172.21.15.78:0/655653109 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870039070 con 0x7f58780aec60 2024-04-15T15:15:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f5875d9b640 1 -- 172.21.15.78:0/655653109 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f587002cc60 con 0x7f58780aec60 2024-04-15T15:15:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f5875d9b640 1 -- 172.21.15.78:0/655653109 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870034660 con 0x7f58780aec60 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f587cd50640 1 -- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58780aec60 msgr2=0x7f58780af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f587cd50640 1 --2- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58780aec60 0x7f58780af0f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5870009d90 tx=0x7f587002beb0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f587cd50640 1 -- 172.21.15.78:0/655653109 shutdown_connections 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.009+0000 7f587cd50640 1 --2- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f58780aec60 0x7f58780af0f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.010+0000 7f587cd50640 1 --2- 172.21.15.78:0/655653109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f58780b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.010+0000 7f587cd50640 1 -- 172.21.15.78:0/655653109 >> 172.21.15.78:0/655653109 conn(0x7f58780b6720 msgr2=0x7f58780b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.010+0000 7f587cd50640 1 -- 172.21.15.78:0/655653109 shutdown_connections 2024-04-15T15:15:15.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.010+0000 7f587cd50640 1 -- 172.21.15.78:0/655653109 wait complete. 2024-04-15T15:15:15.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.011+0000 7f587cd50640 1 Processor -- start 2024-04-15T15:15:15.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.011+0000 7f587cd50640 1 -- start start 2024-04-15T15:15:15.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587cd50640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58780aec60 0x7f5878142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587cd50640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f5878153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587cd50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 0x7f5878143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587cd50640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f587815b940 con 0x7f58780aec60 2024-04-15T15:15:15.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587cd50640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f58780af630 con 0x7f587815b560 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587cd50640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5878152d90 con 0x7f58780b00a0 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f5876d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 0x7f5878143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f5878153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f5878153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53872/0 (socket says 172.21.15.78:53872) 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f5876d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 0x7f5878143220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40392/0 (socket says 172.21.15.78:40392) 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 -- 172.21.15.78:0/4071830068 learned_addr learned my addr 172.21.15.78:0/4071830068 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f587659c640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58780aec60 0x7f5878142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 -- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 msgr2=0x7f5878143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 0x7f5878143220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 -- 172.21.15.78:0/4071830068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58780aec60 msgr2=0x7f5878142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58780aec60 0x7f5878142b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 -- 172.21.15.78:0/4071830068 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58781524a0 con 0x7f58780b00a0 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f5876d9d640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 0x7f5878143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:15.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.012+0000 7f586dd9b640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f5878153220 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f586000e9e0 tx=0x7f586000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5860017070 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f586000c9d0 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5878145f70 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586000cdc0 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5878152a20 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f586001a070 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.013+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f5860013820 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.014+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58780690f0 con 0x7f58780b00a0 2024-04-15T15:15:15.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.016+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f58600066b0 con 0x7f58780b00a0 2024-04-15T15:15:15.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.088+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e6b51350-f913-4f16-ab9f-dbb94a6279ed", "id": 1} v 0) -- 0x7f58780753c0 con 0x7f58780b00a0 2024-04-15T15:15:15.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.091+0000 7f586f7fe640 1 -- 172.21.15.78:0/4071830068 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e6b51350-f913-4f16-ab9f-dbb94a6279ed", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f5860006020 con 0x7f58780b00a0 2024-04-15T15:15:15.093 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 msgr2=0x7f5878153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f5878153220 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f586000e9e0 tx=0x7f586000eea0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 shutdown_connections 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587815b560 0x7f5878143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f58780b00a0 0x7f5878153220 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 --2- 172.21.15.78:0/4071830068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58780aec60 0x7f5878142b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 >> 172.21.15.78:0/4071830068 conn(0x7f58780b6720 msgr2=0x7f58780b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.092+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 shutdown_connections 2024-04-15T15:15:15.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.093+0000 7f587cd50640 1 -- 172.21.15.78:0/4071830068 wait complete. 2024-04-15T15:15:15.105 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new 04a46859-f5e6-441d-81ba-5245caffd22b 2 2024-04-15T15:15:15.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.206+0000 7f1c1129a640 1 Processor -- start 2024-04-15T15:15:15.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.206+0000 7f1c1129a640 1 -- start start 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c1129a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c0c0b09f0 0x7f1c0c0aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c1129a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0af420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c1129a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c0c0ae440 con 0x7f1c0c0b1240 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c1129a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f1c0c06c590 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c1129a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c0ae6e0 con 0x7f1c0c0b6ff0 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c0c0b09f0 0x7f1c0c0aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1c0c0b09f0 0x7f1c0c0aeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40402/0 (socket says 172.21.15.78:40402) 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 -- 172.21.15.78:0/3590945047 learned_addr learned my addr 172.21.15.78:0/3590945047 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0b7fe640 1 --2- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 -- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b6ff0 msgr2=0x7f1c0c0af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 --2- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0af420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 -- 172.21.15.78:0/3590945047 >> v1:172.21.15.46:6789/0 conn(0x7f1c0c0b1240 legacy=0x7f1c0c0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.207+0000 7f1c0a7fc640 1 -- 172.21.15.78:0/3590945047 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c06c870 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.211 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.208+0000 7f4a1243e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:15:15.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.208+0000 7f1c0a7fc640 1 --2- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b09f0 0x7f1c0c0aeee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009870 tx=0x7f1bf402ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08f6a198cc30de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.208+0000 7f1c09ffb640 1 -- 172.21.15.78:0/3590945047 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf402f830 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.208+0000 7f1c09ffb640 1 -- 172.21.15.78:0/3590945047 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf402f9d0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.208+0000 7f1c09ffb640 1 -- 172.21.15.78:0/3590945047 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf402fda0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 -- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b09f0 msgr2=0x7f1c0c0aeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 --2- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b09f0 0x7f1c0c0aeee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009870 tx=0x7f1bf402ba10 comp rx=0 tx=0).stop 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 -- 172.21.15.78:0/3590945047 shutdown_connections 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 --2- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0af420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 --2- 172.21.15.78:0/3590945047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b09f0 0x7f1c0c0aeee0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 -- 172.21.15.78:0/3590945047 >> 172.21.15.78:0/3590945047 conn(0x7f1c0c09a9d0 msgr2=0x7f1c0c0b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 -- 172.21.15.78:0/3590945047 shutdown_connections 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.209+0000 7f1c1129a640 1 -- 172.21.15.78:0/3590945047 wait complete. 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.210+0000 7f1c1129a640 1 Processor -- start 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.210+0000 7f1c1129a640 1 -- start start 2024-04-15T15:15:15.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c1129a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b09f0 0x7f1c0c0c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c1129a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b1240 0x7f1c0c0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c1129a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c1129a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c0afae0 con 0x7f1c0c0b6ff0 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c1129a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0c0af960 con 0x7f1c0c0b1240 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c1129a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0c0afc60 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b1240 0x7f1c0c0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0affd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b09f0 0x7f1c0c0c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0a7fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b1240 0x7f1c0c0ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40408/0 (socket says 172.21.15.78:40408) 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0a7fc640 1 -- 172.21.15.78:0/430940811 learned_addr learned my addr 172.21.15.78:0/430940811 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0b7fe640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0affd640 1 -- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b1240 msgr2=0x7f1c0c0ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.211+0000 7f1c0affd640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b1240 0x7f1c0c0ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c0affd640 1 -- 172.21.15.78:0/430940811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c0c0b6ff0 msgr2=0x7f1c0c0c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c0affd640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c0affd640 1 -- 172.21.15.78:0/430940811 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0c09c6a0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c0affd640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b09f0 0x7f1c0c0c8720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00c340 tx=0x7f1bfc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0052d0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bfc005490 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0c09c980 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc007260 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0c0c8c60 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1bfc007400 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.212+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f1bfc011590 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.213+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c0c0ae9c0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.217+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1c0c0ae9c0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.223 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.222+0000 7fda0e23b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:15:15.267 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.266+0000 7f4a1243e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:15:15.280 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.278+0000 7fda0e23b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:15:15.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.289+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd new", "uuid": "04a46859-f5e6-441d-81ba-5245caffd22b", "id": 2} v 0) -- 0x7f1c0c0690f0 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.292+0000 7f1c037fe640 1 -- 172.21.15.78:0/430940811 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04a46859-f5e6-441d-81ba-5245caffd22b", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f1bfc003a50 con 0x7f1c0c0b09f0 2024-04-15T15:15:15.294 INFO:teuthology.orchestra.run.smithi078.stdout:2 2024-04-15T15:15:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b09f0 msgr2=0x7f1c0c0c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b09f0 0x7f1c0c0c8720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00c340 tx=0x7f1bfc00c800 comp rx=0 tx=0).stop 2024-04-15T15:15:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 shutdown_connections 2024-04-15T15:15:15.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c0c0b6ff0 0x7f1c0c0c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c0c0b1240 0x7f1c0c0ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 --2- 172.21.15.78:0/430940811 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1c0c0b09f0 0x7f1c0c0c8720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 >> 172.21.15.78:0/430940811 conn(0x7f1c0c09a9d0 msgr2=0x7f1c0c070340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 shutdown_connections 2024-04-15T15:15:15.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.293+0000 7f1c1129a640 1 -- 172.21.15.78:0/430940811 wait complete. 2024-04-15T15:15:15.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new 6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0 3 2024-04-15T15:15:15.315 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.314+0000 7f4a1243e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:15:15.326 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.325+0000 7fda0e23b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:15:15.373 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.371+0000 7f4a1243e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:15:15.382 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.380+0000 7fda0e23b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:15:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.407+0000 7f2c2bd75640 1 Processor -- start 2024-04-15T15:15:15.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.407+0000 7f2c2bd75640 1 -- start start 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2bd75640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c2415b520 0x7f2c240ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2bd75640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cf650 0x7f2c240cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2bd75640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c240cbee0 con 0x7f2c240cee70 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2bd75640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2c240cc1c0 con 0x7f2c2415b520 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2bd75640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2c240cc4a0 con 0x7f2c240cf650 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c2415b520 0x7f2c240ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2a312640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cf650 0x7f2c240cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c2415b520 0x7f2c240ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40414/0 (socket says 172.21.15.78:40414) 2024-04-15T15:15:15.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29310640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2c240cee70 0x7f2c240ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40324/0 (socket says 172.21.15.78:40324) 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 -- 172.21.15.78:0/1706694466 learned_addr learned my addr 172.21.15.78:0/1706694466 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 -- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cf650 msgr2=0x7f2c240cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 --2- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cf650 0x7f2c240cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 -- 172.21.15.78:0/1706694466 >> v1:172.21.15.78:6789/0 conn(0x7f2c240cee70 legacy=0x7f2c240ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c28b0f640 1 -- 172.21.15.78:0/1706694466 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084611820 0 0) 0x7f2c240cbee0 con 0x7f2c240cee70 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c29b11640 1 -- 172.21.15.78:0/1706694466 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c240cc780 con 0x7f2c2415b520 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.408+0000 7f2c2a312640 1 --2- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cf650 0x7f2c240cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.409+0000 7f2c29b11640 1 --2- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c2415b520 0x7f2c240ccca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009870 tx=0x7f2c2002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9198523908471d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.409+0000 7f2c28b0f640 1 -- 172.21.15.78:0/1706694466 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20034070 con 0x7f2c2415b520 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.409+0000 7f2c28b0f640 1 -- 172.21.15.78:0/1706694466 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c20030840 con 0x7f2c2415b520 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.409+0000 7f2c28b0f640 1 -- 172.21.15.78:0/1706694466 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20030c10 con 0x7f2c2415b520 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 -- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c2415b520 msgr2=0x7f2c240ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c2415b520 0x7f2c240ccca0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009870 tx=0x7f2c2002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 -- 172.21.15.78:0/1706694466 shutdown_connections 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cf650 0x7f2c240cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/1706694466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c2415b520 0x7f2c240ccca0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 -- 172.21.15.78:0/1706694466 >> 172.21.15.78:0/1706694466 conn(0x7f2c240b2e70 msgr2=0x7f2c240b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 -- 172.21.15.78:0/1706694466 shutdown_connections 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.410+0000 7f2c2bd75640 1 -- 172.21.15.78:0/1706694466 wait complete. 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.411+0000 7f2c2bd75640 1 Processor -- start 2024-04-15T15:15:15.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.411+0000 7f2c2bd75640 1 -- start start 2024-04-15T15:15:15.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2bd75640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 0x7f2c240c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2bd75640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c240cf650 0x7f2c240ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2bd75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415b520 0x7f2c241f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2bd75640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2415b900 con 0x7f2c2415b520 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2bd75640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2406eda0 con 0x7f2c240cf650 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2bd75640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2c240caea0 con 0x7f2c240cee70 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 0x7f2c240c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 0x7f2c240c4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53908/0 (socket says 172.21.15.78:53908) 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 -- 172.21.15.78:0/102093511 learned_addr learned my addr 172.21.15.78:0/102093511 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29310640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c240cf650 0x7f2c240ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2a312640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415b520 0x7f2c241f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 -- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c240cf650 msgr2=0x7f2c240ca960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c240cf650 0x7f2c240ca960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 -- 172.21.15.78:0/102093511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415b520 msgr2=0x7f2c241f85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415b520 0x7f2c241f85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c29b11640 1 -- 172.21.15.78:0/102093511 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c240c6ad0 con 0x7f2c240cee70 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.412+0000 7f2c2a312640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415b520 0x7f2c241f85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c29b11640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 0x7f2c240c4dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009870 tx=0x7f2c200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20034040 con 0x7f2c240cee70 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c2003ab90 con 0x7f2c240cee70 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c20004030 con 0x7f2c240cee70 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c240c6d90 con 0x7f2c240cee70 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c240c7350 con 0x7f2c240cee70 2024-04-15T15:15:15.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2c2004b060 con 0x7f2c240cee70 2024-04-15T15:15:15.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.413+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f2c20003d20 con 0x7f2c240cee70 2024-04-15T15:15:15.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.414+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c240690f0 con 0x7f2c240cee70 2024-04-15T15:15:15.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.416+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2c2003f070 con 0x7f2c240cee70 2024-04-15T15:15:15.420 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.416+0000 7f4a1243e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:15:15.426 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.425+0000 7fda0e23b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:15:15.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.488+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0", "id": 3} v 0) -- 0x7f2c240cfd70 con 0x7f2c240cee70 2024-04-15T15:15:15.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.491+0000 7f2c167fc640 1 -- 172.21.15.78:0/102093511 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f2c20003890 con 0x7f2c240cee70 2024-04-15T15:15:15.493 INFO:teuthology.orchestra.run.smithi078.stdout:3 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 msgr2=0x7f2c240c4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 0x7f2c240c4dd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2c20009870 tx=0x7f2c200046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 shutdown_connections 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c2415b520 0x7f2c241f85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c240cf650 0x7f2c240ca960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 --2- 172.21.15.78:0/102093511 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c240cee70 0x7f2c240c4dd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 >> 172.21.15.78:0/102093511 conn(0x7f2c240b2e70 msgr2=0x7f2c240b26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 shutdown_connections 2024-04-15T15:15:15.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.492+0000 7f2c2bd75640 1 -- 172.21.15.78:0/102093511 wait complete. 2024-04-15T15:15:15.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new 0192cc67-2b52-4c62-882f-5ccd13326bb1 4 2024-04-15T15:15:15.542 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.538+0000 7f4a1243e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:15:15.548 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.546+0000 7fda0e23b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:15:15.614 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.612+0000 7f4a1243e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:15:15.621 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.619+0000 7fda0e23b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:15:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.643+0000 7fb2902c5640 1 Processor -- start 2024-04-15T15:15:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.643+0000 7fb2902c5640 1 -- start start 2024-04-15T15:15:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb2902c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2880b1240 0x7fb2880b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb2902c5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880afef0 0x7fb2880ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb2902c5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2880ae440 con 0x7fb2880b09f0 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb2902c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb2880b1b60 con 0x7fb2880b1240 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb2902c5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb28806c590 con 0x7fb2880afef0 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28e862640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880afef0 0x7fb2880ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28d860640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb2880b09f0 0x7fb2880cdda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40326/0 (socket says 172.21.15.78:40326) 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28d860640 1 -- 172.21.15.78:0/2814399244 learned_addr learned my addr 172.21.15.78:0/2814399244 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28e061640 1 --2- 172.21.15.78:0/2814399244 >> v2:172.21.15.46:3300/0 conn(0x7fb2880b1240 0x7fb2880b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28d05f640 1 -- 172.21.15.78:0/2814399244 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522614316 0 0) 0x7fb2880ae440 con 0x7fb2880b09f0 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28d05f640 1 -- 172.21.15.78:0/2814399244 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb274003520 con 0x7fb2880b09f0 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28e862640 1 -- 172.21.15.78:0/2814399244 >> v2:172.21.15.46:3300/0 conn(0x7fb2880b1240 msgr2=0x7fb2880b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.644+0000 7fb28e862640 1 --2- 172.21.15.78:0/2814399244 >> v2:172.21.15.46:3300/0 conn(0x7fb2880b1240 0x7fb2880b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.645+0000 7fb28e862640 1 -- 172.21.15.78:0/2814399244 >> v1:172.21.15.78:6789/0 conn(0x7fb2880b09f0 legacy=0x7fb2880cdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.645+0000 7fb28e862640 1 -- 172.21.15.78:0/2814399244 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2880ae6e0 con 0x7fb2880afef0 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.645+0000 7fb28e862640 1 --2- 172.21.15.78:0/2814399244 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880afef0 0x7fb2880ce4b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb280009d90 tx=0x7fb28002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea287b0066bf231 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.645+0000 7fb28d05f640 1 -- 172.21.15.78:0/2814399244 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280039070 con 0x7fb2880afef0 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.645+0000 7fb28d05f640 1 -- 172.21.15.78:0/2814399244 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb28002cce0 con 0x7fb2880afef0 2024-04-15T15:15:15.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.645+0000 7fb28d05f640 1 -- 172.21.15.78:0/2814399244 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280034660 con 0x7fb2880afef0 2024-04-15T15:15:15.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 -- 172.21.15.78:0/2814399244 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880afef0 msgr2=0x7fb2880ce4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 --2- 172.21.15.78:0/2814399244 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880afef0 0x7fb2880ce4b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb280009d90 tx=0x7fb28002bf30 comp rx=0 tx=0).stop 2024-04-15T15:15:15.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 -- 172.21.15.78:0/2814399244 shutdown_connections 2024-04-15T15:15:15.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 --2- 172.21.15.78:0/2814399244 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880afef0 0x7fb2880ce4b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 --2- 172.21.15.78:0/2814399244 >> v2:172.21.15.46:3300/0 conn(0x7fb2880b1240 0x7fb2880b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 -- 172.21.15.78:0/2814399244 >> 172.21.15.78:0/2814399244 conn(0x7fb28809a9d0 msgr2=0x7fb2880b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 -- 172.21.15.78:0/2814399244 shutdown_connections 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.646+0000 7fb2902c5640 1 -- 172.21.15.78:0/2814399244 wait complete. 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.647+0000 7fb2902c5640 1 Processor -- start 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.647+0000 7fb2902c5640 1 -- start start 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb2902c5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 0x7fb28814cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb2902c5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880b09f0 0x7fb28814b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb2902c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2880b1240 0x7fb28814bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb2902c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2880ce9f0 con 0x7fb2880b1240 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb2902c5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb2880b1f30 con 0x7fb2880afef0 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb2902c5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb2880ceb70 con 0x7fb2880b09f0 2024-04-15T15:15:15.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 0x7fb28814cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28d860640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880b09f0 0x7fb28814b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 0x7fb28814cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40430/0 (socket says 172.21.15.78:40430) 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28d860640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880b09f0 0x7fb28814b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53916/0 (socket says 172.21.15.78:53916) 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 -- 172.21.15.78:0/459557427 learned_addr learned my addr 172.21.15.78:0/459557427 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e862640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2880b1240 0x7fb28814bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 -- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880b09f0 msgr2=0x7fb28814b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880b09f0 0x7fb28814b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 -- 172.21.15.78:0/459557427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2880b1240 msgr2=0x7fb28814bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2880b1240 0x7fb28814bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.648+0000 7fb28e061640 1 -- 172.21.15.78:0/459557427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb288149640 con 0x7fb2880afef0 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb28e061640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 0x7fb28814cb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb27800e930 tx=0x7fb27800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb278017070 con 0x7fb2880afef0 2024-04-15T15:15:15.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27800ca00 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27800cdf0 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb288149920 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb288149ee0 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb27801a070 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.649+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fb27801caf0 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.650+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2880690f0 con 0x7fb2880afef0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.652+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb27801c5e0 con 0x7fb2880afef0 2024-04-15T15:15:15.663 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.661+0000 7f4a1243e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:15:15.669 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.667+0000 7fda0e23b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:15:15.717 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.716+0000 7f4a1243e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:15:15.723 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.722+0000 7fda0e23b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:15:15.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.725+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0192cc67-2b52-4c62-882f-5ccd13326bb1", "id": 4} v 0) -- 0x7fb2880cf8c0 con 0x7fb2880afef0 2024-04-15T15:15:15.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.728+0000 7fb27effd640 1 -- 172.21.15.78:0/459557427 <== mon.1 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0192cc67-2b52-4c62-882f-5ccd13326bb1", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fb27801c7c0 con 0x7fb2880afef0 2024-04-15T15:15:15.730 INFO:teuthology.orchestra.run.smithi078.stdout:4 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 msgr2=0x7fb28814cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 0x7fb28814cb30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb27800e930 tx=0x7fb27800edf0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 shutdown_connections 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2880b1240 0x7fb28814bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2880b09f0 0x7fb28814b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 --2- 172.21.15.78:0/459557427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2880afef0 0x7fb28814cb30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 >> 172.21.15.78:0/459557427 conn(0x7fb28809a9d0 msgr2=0x7fb28806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 shutdown_connections 2024-04-15T15:15:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.729+0000 7fb2902c5640 1 -- 172.21.15.78:0/459557427 wait complete. 2024-04-15T15:15:15.741 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new 821d1a3b-75b6-4850-9511-84b280133a6e 5 2024-04-15T15:15:15.768 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:15.767+0000 7f4a1243e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:15:15.773 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:15.771+0000 7fda0e23b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:15:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.842+0000 7fc199b52640 1 Processor -- start 2024-04-15T15:15:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.842+0000 7fc199b52640 1 -- start start 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc199b52640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc19415eb00 0x7fc1940ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc199b52640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 0x7fc19406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc199b52640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1940cbd10 con 0x7fc1940b09a0 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc199b52640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc19415eee0 con 0x7fc19415eb00 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc199b52640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc1940cbff0 con 0x7fc1940b0560 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 0x7fc19406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 0x7fc19406b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53920/0 (socket says 172.21.15.78:53920) 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 -- 172.21.15.78:0/2453924769 learned_addr learned my addr 172.21.15.78:0/2453924769 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc1937fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc19415eb00 0x7fc1940ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 -- 172.21.15.78:0/2453924769 >> v2:172.21.15.78:3300/0 conn(0x7fc19415eb00 msgr2=0x7fc1940ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 --2- 172.21.15.78:0/2453924769 >> v2:172.21.15.78:3300/0 conn(0x7fc19415eb00 0x7fc1940ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 -- 172.21.15.78:0/2453924769 >> v1:172.21.15.46:6789/0 conn(0x7fc1940b09a0 legacy=0x7fc1940ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc18affd640 1 -- 172.21.15.78:0/2453924769 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1940cc2d0 con 0x7fc1940b0560 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.843+0000 7fc1937fe640 1 --2- 172.21.15.78:0/2453924769 >> v2:172.21.15.78:3300/0 conn(0x7fc19415eb00 0x7fc1940ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.844+0000 7fc18affd640 1 --2- 172.21.15.78:0/2453924769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 0x7fc19406b9e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009870 tx=0x7fc17c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2f01889f5f86a37 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.844+0000 7fc192ffd640 1 -- 172.21.15.78:0/2453924769 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c034070 con 0x7fc1940b0560 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.844+0000 7fc192ffd640 1 -- 172.21.15.78:0/2453924769 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc17c030840 con 0x7fc1940b0560 2024-04-15T15:15:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.844+0000 7fc192ffd640 1 -- 172.21.15.78:0/2453924769 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c030c10 con 0x7fc1940b0560 2024-04-15T15:15:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 -- 172.21.15.78:0/2453924769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 msgr2=0x7fc19406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 --2- 172.21.15.78:0/2453924769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 0x7fc19406b9e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009870 tx=0x7fc17c02b980 comp rx=0 tx=0).stop 2024-04-15T15:15:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 -- 172.21.15.78:0/2453924769 shutdown_connections 2024-04-15T15:15:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 --2- 172.21.15.78:0/2453924769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b0560 0x7fc19406b9e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 --2- 172.21.15.78:0/2453924769 >> v2:172.21.15.78:3300/0 conn(0x7fc19415eb00 0x7fc1940ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 -- 172.21.15.78:0/2453924769 >> 172.21.15.78:0/2453924769 conn(0x7fc19409a9d0 msgr2=0x7fc1940d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 -- 172.21.15.78:0/2453924769 shutdown_connections 2024-04-15T15:15:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.845+0000 7fc199b52640 1 -- 172.21.15.78:0/2453924769 wait complete. 2024-04-15T15:15:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.847+0000 7fc199b52640 1 Processor -- start 2024-04-15T15:15:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.847+0000 7fc199b52640 1 -- start start 2024-04-15T15:15:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc199b52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1940b0560 0x7fc194152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc199b52640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 0x7fc1940b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc199b52640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19415eb00 0x7fc19414f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc199b52640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940ae7e0 con 0x7fc1940b0560 2024-04-15T15:15:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc199b52640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc1940cc890 con 0x7fc19415eb00 2024-04-15T15:15:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc199b52640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc1940cec60 con 0x7fc1940b09a0 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc18affd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 0x7fc1940b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc193fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19415eb00 0x7fc19414f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.848+0000 7fc18affd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 0x7fc1940b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53922/0 (socket says 172.21.15.78:53922) 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 -- 172.21.15.78:0/2001220766 learned_addr learned my addr 172.21.15.78:0/2001220766 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc1937fe640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1940b0560 0x7fc194152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 -- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19415eb00 msgr2=0x7fc19414f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19415eb00 0x7fc19414f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 -- 172.21.15.78:0/2001220766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1940b0560 msgr2=0x7fc194152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1940b0560 0x7fc194152e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 -- 172.21.15.78:0/2001220766 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1940ca1f0 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc1937fe640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1940b0560 0x7fc194152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc193fff640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19415eb00 0x7fc19414f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc18affd640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 0x7fc1940b4190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc17c000c00 tx=0x7fc17c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c034040 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc17c039d90 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1940ca490 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c043610 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1940caa50 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc17c04a060 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.849+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fc17c02f910 con 0x7fc1940b09a0 2024-04-15T15:15:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.850+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1940690f0 con 0x7fc1940b09a0 2024-04-15T15:15:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.852+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc17c062e50 con 0x7fc1940b09a0 2024-04-15T15:15:15.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.924+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd new", "uuid": "821d1a3b-75b6-4850-9511-84b280133a6e", "id": 5} v 0) -- 0x7fc1940cf5b0 con 0x7fc1940b09a0 2024-04-15T15:15:15.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.927+0000 7fc190ff9640 1 -- 172.21.15.78:0/2001220766 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "821d1a3b-75b6-4850-9511-84b280133a6e", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fc17c051030 con 0x7fc1940b09a0 2024-04-15T15:15:15.929 INFO:teuthology.orchestra.run.smithi078.stdout:5 2024-04-15T15:15:15.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 msgr2=0x7fc1940b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 0x7fc1940b4190 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc17c000c00 tx=0x7fc17c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 shutdown_connections 2024-04-15T15:15:15.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc19415eb00 0x7fc19414f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc1940b09a0 0x7fc1940b4190 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 --2- 172.21.15.78:0/2001220766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1940b0560 0x7fc194152e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 >> 172.21.15.78:0/2001220766 conn(0x7fc19409a9d0 msgr2=0x7fc19406ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 shutdown_connections 2024-04-15T15:15:15.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:15.928+0000 7fc199b52640 1 -- 172.21.15.78:0/2001220766 wait complete. 2024-04-15T15:15:15.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new 38d4f697-6404-4d5a-96fe-d1e6dd1933be 6 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.082+0000 7f9fba6b2640 1 Processor -- start 2024-04-15T15:15:16.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.082+0000 7f9fba6b2640 1 -- start start 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fba6b2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb4070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fba6b2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 0x7f9fb40cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fba6b2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb40ae440 con 0x7f9fb406ef70 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fba6b2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f9fb40cffa0 con 0x7f9fb40cf600 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fba6b2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb40ae720 con 0x7f9fb415ed00 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 0x7f9fb40cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb4070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb37fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 0x7f9fb40cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40456/0 (socket says 172.21.15.78:40456) 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb4070920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53928/0 (socket says 172.21.15.78:53928) 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb37fe640 1 -- 172.21.15.78:0/2067600605 learned_addr learned my addr 172.21.15.78:0/2067600605 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb3fff640 1 -- 172.21.15.78:0/2067600605 >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 msgr2=0x7f9fb40cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb3fff640 1 --2- 172.21.15.78:0/2067600605 >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 0x7f9fb40cfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb3fff640 1 -- 172.21.15.78:0/2067600605 >> v1:172.21.15.46:6789/0 conn(0x7f9fb406ef70 legacy=0x7f9fb406f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb3fff640 1 -- 172.21.15.78:0/2067600605 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb40aea00 con 0x7f9fb415ed00 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.083+0000 7f9fb37fe640 1 --2- 172.21.15.78:0/2067600605 >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 0x7f9fb40cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.084+0000 7f9fb3fff640 1 --2- 172.21.15.78:0/2067600605 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb4070920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40097a0 tx=0x7f9fa402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d708e97ab810db9e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.084+0000 7f9fb2ffd640 1 -- 172.21.15.78:0/2067600605 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4035020 con 0x7f9fb415ed00 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.084+0000 7f9fb2ffd640 1 -- 172.21.15.78:0/2067600605 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa4031830 con 0x7f9fb415ed00 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.084+0000 7f9fb2ffd640 1 -- 172.21.15.78:0/2067600605 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa4031c00 con 0x7f9fb415ed00 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.084+0000 7f9fba6b2640 1 -- 172.21.15.78:0/2067600605 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 msgr2=0x7f9fb4070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.084+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/2067600605 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb4070920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40097a0 tx=0x7f9fa402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.085+0000 7f9fba6b2640 1 -- 172.21.15.78:0/2067600605 shutdown_connections 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.085+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/2067600605 >> v2:172.21.15.78:3300/0 conn(0x7f9fb40cf600 0x7f9fb40cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.085+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/2067600605 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb4070920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.085+0000 7f9fba6b2640 1 -- 172.21.15.78:0/2067600605 >> 172.21.15.78:0/2067600605 conn(0x7f9fb409a9d0 msgr2=0x7f9fb40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.085+0000 7f9fba6b2640 1 -- 172.21.15.78:0/2067600605 shutdown_connections 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.085+0000 7f9fba6b2640 1 -- 172.21.15.78:0/2067600605 wait complete. 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.086+0000 7f9fba6b2640 1 Processor -- start 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.086+0000 7f9fba6b2640 1 -- start start 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb406ef70 0x7f9fb40b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 0x7f9fb41523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb40c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb40701a0 con 0x7f9fb406ef70 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb40aefc0 con 0x7f9fb40cf600 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb4070e60 con 0x7f9fb415ed00 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 0x7f9fb41523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb8c4f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb40c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 0x7f9fb41523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40460/0 (socket says 172.21.15.78:40460) 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 -- 172.21.15.78:0/1718974736 learned_addr learned my addr 172.21.15.78:0/1718974736 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb8c4f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb40c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53942/0 (socket says 172.21.15.78:53942) 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb3fff640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb406ef70 0x7f9fb40b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 -- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 msgr2=0x7f9fb40c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb40c50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 -- 172.21.15.78:0/1718974736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb406ef70 msgr2=0x7f9fb40b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb406ef70 0x7f9fb40b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 -- 172.21.15.78:0/1718974736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb4152900 con 0x7f9fb40cf600 2024-04-15T15:15:16.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb37fe640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 0x7f9fb41523c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8010dd0 tx=0x7f9fa800e730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa801e070 con 0x7f9fb40cf600 2024-04-15T15:15:16.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa8011860 con 0x7f9fb40cf600 2024-04-15T15:15:16.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8011c10 con 0x7f9fb40cf600 2024-04-15T15:15:16.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.087+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb40b20a0 con 0x7f9fb40cf600 2024-04-15T15:15:16.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.088+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb40b2620 con 0x7f9fb40cf600 2024-04-15T15:15:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.088+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9fa8021070 con 0x7f9fb40cf600 2024-04-15T15:15:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.088+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f9fa80067a0 con 0x7f9fb40cf600 2024-04-15T15:15:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.088+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb40690f0 con 0x7f9fb40cf600 2024-04-15T15:15:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.091+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9fa8028070 con 0x7f9fb40cf600 2024-04-15T15:15:16.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.164+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd new", "uuid": "38d4f697-6404-4d5a-96fe-d1e6dd1933be", "id": 6} v 0) -- 0x7f9fb4153040 con 0x7f9fb40cf600 2024-04-15T15:15:16.168 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.166+0000 7fda0e23b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:15:16.169 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.167+0000 7f4a1243e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:15:16.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.168+0000 7f9fb0ff9640 1 -- 172.21.15.78:0/1718974736 <== mon.1 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38d4f697-6404-4d5a-96fe-d1e6dd1933be", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f9fa8011db0 con 0x7f9fb40cf600 2024-04-15T15:15:16.170 INFO:teuthology.orchestra.run.smithi078.stdout:6 2024-04-15T15:15:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 msgr2=0x7f9fb41523c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 0x7f9fb41523c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8010dd0 tx=0x7f9fa800e730 comp rx=0 tx=0).stop 2024-04-15T15:15:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 shutdown_connections 2024-04-15T15:15:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fb415ed00 0x7f9fb40c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fb40cf600 0x7f9fb41523c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 --2- 172.21.15.78:0/1718974736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fb406ef70 0x7f9fb40b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 >> 172.21.15.78:0/1718974736 conn(0x7f9fb409a9d0 msgr2=0x7f9fb40ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 shutdown_connections 2024-04-15T15:15:16.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.169+0000 7f9fba6b2640 1 -- 172.21.15.78:0/1718974736 wait complete. 2024-04-15T15:15:16.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd new 9c0bac96-a0f3-4877-868b-dd55bad4a100 7 2024-04-15T15:15:16.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.283+0000 7fb10b150640 1 Processor -- start 2024-04-15T15:15:16.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.283+0000 7fb10b150640 1 -- start start 2024-04-15T15:15:16.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb10b150640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1040b5de0 0x7fb104169430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb10b150640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10406f220 con 0x7fb10415e6f0 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb10b150640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10406f500 con 0x7fb10406d100 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb10b150640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb1040b45d0 con 0x7fb1040b5de0 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb103fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb10406d100 0x7fb10406e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:40342/0 (socket says 172.21.15.78:40342) 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb103fff640 1 -- 172.21.15.78:0/3163054559 learned_addr learned my addr 172.21.15.78:0/3163054559 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1096ed640 1 --2- 172.21.15.78:0/3163054559 >> v2:172.21.15.46:3300/0 conn(0x7fb1040b5de0 0x7fb104169430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2110986478 0 0) 0x7fb10406f500 con 0x7fb10406d100 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0f4003520 con 0x7fb10406d100 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477634532 0 0) 0x7fb10406f220 con 0x7fb10415e6f0 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb10406f500 con 0x7fb10415e6f0 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.1 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1917105932 0 0) 0x7fb0f4003520 con 0x7fb10406d100 2024-04-15T15:15:16.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 >> v2:172.21.15.46:3300/0 conn(0x7fb1040b5de0 msgr2=0x7fb104169430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 --2- 172.21.15.78:0/3163054559 >> v2:172.21.15.46:3300/0 conn(0x7fb1040b5de0 0x7fb104169430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 >> v1:172.21.15.46:6789/0 conn(0x7fb10415e6f0 legacy=0x7fb10406c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1040b48b0 con 0x7fb10406d100 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.1 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fb0f0001670 con 0x7fb10406d100 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 >> v1:172.21.15.78:6789/0 conn(0x7fb10406d100 legacy=0x7fb10406e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f4003b90 0x7fb0f4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0f4004800 0x7fb0f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f40051c0 0x7fb0f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1040b45d0 con 0x7fb0f4004800 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f400c090 con 0x7fb0f40051c0 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f400c290 con 0x7fb0f4003b90 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.284+0000 7fb108eec640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f4003b90 0x7fb0f4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f40051c0 0x7fb0f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb103fff640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0f4004800 0x7fb0f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 -- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f4003b90 msgr2=0x7fb0f4003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f4003b90 0x7fb0f4003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 -- 172.21.15.78:0/3163054559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0f4004800 msgr2=0x7fb0f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb0f4004800 0x7fb0f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 -- 172.21.15.78:0/3163054559 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f4003520 con 0x7fb0f40051c0 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.285+0000 7fb1096ed640 1 -- 172.21.15.78:0/3163054559 reap_dead start 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb1096ed640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f40051c0 0x7fb0f400b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009f90 tx=0x7fb0f4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=200cfae2e8231125 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc039070 con 0x7fb0f40051c0 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02cc20 con 0x7fb0f40051c0 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb1037fe640 1 -- 172.21.15.78:0/3163054559 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc009c10 con 0x7fb0f40051c0 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 -- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f40051c0 msgr2=0x7fb0f400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f40051c0 0x7fb0f400b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009f90 tx=0x7fb0f4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 -- 172.21.15.78:0/3163054559 shutdown_connections 2024-04-15T15:15:16.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 --2- 172.21.15.78:0/3163054559 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb0f40051c0 0x7fb0f400b9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 -- 172.21.15.78:0/3163054559 >> 172.21.15.78:0/3163054559 conn(0x7fb104072440 msgr2=0x7fb104072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 -- 172.21.15.78:0/3163054559 shutdown_connections 2024-04-15T15:15:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.286+0000 7fb10b150640 1 -- 172.21.15.78:0/3163054559 wait complete. 2024-04-15T15:15:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 Processor -- start 2024-04-15T15:15:16.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 -- start start 2024-04-15T15:15:16.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 0x7fb104158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10415edd0 0x7fb1040ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb10415f230 0x7fb1040add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb10415ec50 con 0x7fb10415edd0 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb10415ead0 con 0x7fb10415f230 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.288+0000 7fb10b150640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb10406c9f0 con 0x7fb0f40051c0 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 0x7fb104158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 0x7fb104158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.78:53962/0 (socket says 172.21.15.78:53962) 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 -- 172.21.15.78:0/1802992705 learned_addr learned my addr 172.21.15.78:0/1802992705 (peer_addr_for_me v2:172.21.15.78:0/0) 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb1096ed640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb10415f230 0x7fb1040add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb103fff640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10415edd0 0x7fb1040ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 -- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb10415f230 msgr2=0x7fb1040add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb10415f230 0x7fb1040add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 -- 172.21.15.78:0/1802992705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10415edd0 msgr2=0x7fb1040ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10415edd0 0x7fb1040ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 -- 172.21.15.78:0/1802992705 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1041e75f0 con 0x7fb0f40051c0 2024-04-15T15:15:16.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb108eec640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 0x7fb104158f90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800f7d0 tx=0x7fb0f800fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f801a070 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f8005320 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10415fb20 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8007270 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1041600e0 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb0f801d070 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.289+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fb0f800dd90 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.290+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1040690f0 con 0x7fb0f40051c0 2024-04-15T15:15:16.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.292+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb0f8007410 con 0x7fb0f40051c0 2024-04-15T15:15:16.309 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.307+0000 7fda0e23b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:15:16.312 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.310+0000 7f4a1243e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:15:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.365+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9c0bac96-a0f3-4877-868b-dd55bad4a100", "id": 7} v 0) -- 0x7fb1040b5a70 con 0x7fb0f40051c0 2024-04-15T15:15:16.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.373+0000 7fb1017fa640 1 -- 172.21.15.78:0/1802992705 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c0bac96-a0f3-4877-868b-dd55bad4a100", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fb0f80075f0 con 0x7fb0f40051c0 2024-04-15T15:15:16.375 INFO:teuthology.orchestra.run.smithi078.stdout:7 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 msgr2=0x7fb104158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 0x7fb104158f90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800f7d0 tx=0x7fb0f800fc90 comp rx=0 tx=0).stop 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 shutdown_connections 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb10415f230 0x7fb1040add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb10415edd0 0x7fb1040ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 --2- 172.21.15.78:0/1802992705 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb0f40051c0 0x7fb104158f90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 >> 172.21.15.78:0/1802992705 conn(0x7fb104072440 msgr2=0x7fb10415c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 shutdown_connections 2024-04-15T15:15:16.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:15:16.374+0000 7fb10b150640 1 -- 172.21.15.78:0/1802992705 wait complete. 2024-04-15T15:15:16.386 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-15T15:15:16.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-15T15:15:16.389 INFO:tasks.ceph.osd.0:Started 2024-04-15T15:15:16.389 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-15T15:15:16.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-15T15:15:16.391 INFO:tasks.ceph.osd.1:Started 2024-04-15T15:15:16.391 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-15T15:15:16.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-15T15:15:16.394 INFO:tasks.ceph.osd.2:Started 2024-04-15T15:15:16.394 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-15T15:15:16.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-15T15:15:16.396 INFO:tasks.ceph.osd.3:Started 2024-04-15T15:15:16.396 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-15T15:15:16.396 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-15T15:15:16.399 INFO:tasks.ceph.osd.4:Started 2024-04-15T15:15:16.400 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-15T15:15:16.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-15T15:15:16.402 INFO:tasks.ceph.osd.5:Started 2024-04-15T15:15:16.403 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-15T15:15:16.403 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-15T15:15:16.405 INFO:tasks.ceph.osd.6:Started 2024-04-15T15:15:16.405 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-15T15:15:16.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-15T15:15:16.410 INFO:tasks.ceph.osd.7:Started 2024-04-15T15:15:16.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:16.435 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.433+0000 7f4a1243e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:15:16.444 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.442+0000 7fda0e23b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:15:16.486 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.485+0000 7f4a1243e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:15:16.514 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.512+0000 7fda0e23b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:15:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.554+0000 7f5d66655640 1 Processor -- start 2024-04-15T15:15:16.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.554+0000 7f5d66655640 1 -- start start 2024-04-15T15:15:16.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d66655640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d60158780 0x7f5d6015bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d66655640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d600b4db0 0x7f5d6006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d66655640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d6015c0c0 con 0x7f5d600b3c30 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d66655640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5d6006fd00 con 0x7f5d600b4db0 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d66655640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d60158b60 con 0x7f5d60158780 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d600b4db0 0x7f5d6006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d600b4db0 0x7f5d6006b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37208/0 (socket says 172.21.15.46:37208) 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 -- 172.21.15.46:0/3197440622 learned_addr learned my addr 172.21.15.46:0/3197440622 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d5ffff640 1 --2- 172.21.15.46:0/3197440622 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d60158780 0x7f5d6015bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 -- 172.21.15.46:0/3197440622 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d60158780 msgr2=0x7f5d6015bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 --2- 172.21.15.46:0/3197440622 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d60158780 0x7f5d6015bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 -- 172.21.15.46:0/3197440622 >> v1:172.21.15.78:6789/0 conn(0x7f5d600b3c30 legacy=0x7f5d600b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d64bf2640 1 -- 172.21.15.46:0/3197440622 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6006f0f0 con 0x7f5d600b4db0 2024-04-15T15:15:16.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.555+0000 7f5d5effd640 1 -- 172.21.15.46:0/3197440622 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652652173 0 0) 0x7f5d54001110 con 0x7f5d600b3c30 2024-04-15T15:15:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.556+0000 7f5d64bf2640 1 --2- 172.21.15.46:0/3197440622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b4db0 0x7f5d6006b9e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800af60 tx=0x7f5d5802cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfa5b38870d0ab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.557+0000 7f5d5effd640 1 -- 172.21.15.46:0/3197440622 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5802d8f0 con 0x7f5d600b4db0 2024-04-15T15:15:16.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.557+0000 7f5d5effd640 1 -- 172.21.15.46:0/3197440622 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5802da90 con 0x7f5d600b4db0 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.557+0000 7f5d5effd640 1 -- 172.21.15.46:0/3197440622 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5802de60 con 0x7f5d600b4db0 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 -- 172.21.15.46:0/3197440622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b4db0 msgr2=0x7f5d6006b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 --2- 172.21.15.46:0/3197440622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b4db0 0x7f5d6006b9e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800af60 tx=0x7f5d5802cf50 comp rx=0 tx=0).stop 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 -- 172.21.15.46:0/3197440622 shutdown_connections 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 --2- 172.21.15.46:0/3197440622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b4db0 0x7f5d6006b9e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 --2- 172.21.15.46:0/3197440622 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d60158780 0x7f5d6015bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 -- 172.21.15.46:0/3197440622 >> 172.21.15.46:0/3197440622 conn(0x7f5d600d0c80 msgr2=0x7f5d600d1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 -- 172.21.15.46:0/3197440622 shutdown_connections 2024-04-15T15:15:16.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.558+0000 7f5d66655640 1 -- 172.21.15.46:0/3197440622 wait complete. 2024-04-15T15:15:16.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.559+0000 7f5d66655640 1 Processor -- start 2024-04-15T15:15:16.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.559+0000 7f5d66655640 1 -- start start 2024-04-15T15:15:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 0x7f5d6015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d600b4db0 0x7f5d601f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d60158780 0x7f5d60148810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6006f990 con 0x7f5d600b3c30 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6006f6b0 con 0x7f5d60158780 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d60070190 con 0x7f5d600b4db0 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 0x7f5d6015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 0x7f5d6015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37224/0 (socket says 172.21.15.46:37224) 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 -- 172.21.15.46:0/732195016 learned_addr learned my addr 172.21.15.46:0/732195016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d64bf2640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d60158780 0x7f5d60148810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5f7fe640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d600b4db0 0x7f5d601f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 -- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d600b4db0 msgr2=0x7f5d601f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d600b4db0 0x7f5d601f7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 -- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d60158780 msgr2=0x7f5d60148810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d60158780 0x7f5d60148810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 -- 172.21.15.46:0/732195016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d60148d50 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5ffff640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 0x7f5d6015a9c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000e870 tx=0x7f5d5000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50017070 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5000c940 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.560+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d601f8270 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.561+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5000cd30 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.561+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d601f8830 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.561+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5d5001a070 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.561+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f5d50006570 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.561+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d600690f0 con 0x7f5d600b3c30 2024-04-15T15:15:16.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.564+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5d60148d50 con 0x7f5d600b3c30 2024-04-15T15:15:16.604 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.602+0000 7f4a1243e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:15:16.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.626+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d6006e910 con 0x7f5d600b3c30 2024-04-15T15:15:16.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d5cff9640 1 -- 172.21.15.46:0/732195016 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f5d50006810 con 0x7f5d600b3c30 2024-04-15T15:15:16.628 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:16.629 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:16.367003+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 msgr2=0x7f5d6015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 0x7f5d6015a9c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000e870 tx=0x7f5d5000ed30 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 shutdown_connections 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d60158780 0x7f5d60148810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d600b4db0 0x7f5d601f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 --2- 172.21.15.46:0/732195016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d600b3c30 0x7f5d6015a9c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.627+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 >> 172.21.15.46:0/732195016 conn(0x7f5d600d0c80 msgr2=0x7f5d6015d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.628+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 shutdown_connections 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:16.628+0000 7f5d66655640 1 -- 172.21.15.46:0/732195016 wait complete. 2024-04-15T15:15:16.641 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-15T15:15:16.641 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-15T15:15:16.648 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.644+0000 7fda0e23b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:15:16.697 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.695+0000 7f4a1243e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:15:16.745 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.744+0000 7fda0e23b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:15:16.853 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.851+0000 7f4a1243e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:15:16.895 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:16.893+0000 7f4a1243e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:15:16.907 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.905+0000 7fda0e23b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:15:16.951 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:16.950+0000 7fda0e23b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:15:16.997 INFO:tasks.ceph.osd.4.smithi078.stderr:2024-04-15T15:15:16.995+0000 7f8d4c8d3640 -1 Falling back to public interface 2024-04-15T15:15:17.005 INFO:tasks.ceph.osd.7.smithi078.stderr:2024-04-15T15:15:17.003+0000 7f7763e1b640 -1 Falling back to public interface 2024-04-15T15:15:17.012 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-04-15T15:15:17.010+0000 7fd895ffa640 -1 Falling back to public interface 2024-04-15T15:15:17.012 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-04-15T15:15:17.011+0000 7f33890a4640 -1 Falling back to public interface 2024-04-15T15:15:17.016 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-04-15T15:15:17.014+0000 7f46c7437640 -1 Falling back to public interface 2024-04-15T15:15:17.016 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-04-15T15:15:17.014+0000 7fe4bdb9e640 -1 Falling back to public interface 2024-04-15T15:15:17.025 INFO:tasks.ceph.osd.5.smithi078.stderr:2024-04-15T15:15:17.023+0000 7f29b3591640 -1 Falling back to public interface 2024-04-15T15:15:17.025 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-04-15T15:15:17.023+0000 7fa2ee391640 -1 Falling back to public interface 2024-04-15T15:15:17.054 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:17.052+0000 7f4a1243e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:15:17.118 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:17.117+0000 7fda0e23b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:15:17.451 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:17.544 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.542+0000 7feb58329640 1 Processor -- start 2024-04-15T15:15:17.544 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.542+0000 7feb58329640 1 -- start start 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.543+0000 7feb58329640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb501459c0 0x7feb500731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.543+0000 7feb58329640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb5006c0f0 0x7feb500b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.543+0000 7feb58329640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb501463a0 con 0x7feb50145f60 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.543+0000 7feb58329640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7feb500b36f0 con 0x7feb5006c0f0 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.543+0000 7feb58329640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7feb50145da0 con 0x7feb501459c0 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb5006c0f0 0x7feb500b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7feb5006c0f0 0x7feb500b06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37232/0 (socket says 172.21.15.46:37232) 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 -- 172.21.15.46:0/275773555 learned_addr learned my addr 172.21.15.46:0/275773555 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:17.546 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb560c5640 1 --2- 172.21.15.46:0/275773555 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb501459c0 0x7feb500731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.547 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 -- 172.21.15.46:0/275773555 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb501459c0 msgr2=0x7feb500731f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.547 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 --2- 172.21.15.46:0/275773555 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb501459c0 0x7feb500731f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.547 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 -- 172.21.15.46:0/275773555 >> v1:172.21.15.78:6789/0 conn(0x7feb50145f60 legacy=0x7feb5006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.547 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.544+0000 7feb568c6640 1 -- 172.21.15.46:0/275773555 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5009aae0 con 0x7feb5006c0f0 2024-04-15T15:15:17.548 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.545+0000 7feb568c6640 1 --2- 172.21.15.46:0/275773555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb5006c0f0 0x7feb500b06e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb40009870 tx=0x7feb4002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31eae86b22d6a66e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.548 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.545+0000 7feb550c3640 1 -- 172.21.15.46:0/275773555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb40034070 con 0x7feb5006c0f0 2024-04-15T15:15:17.548 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.545+0000 7feb550c3640 1 -- 172.21.15.46:0/275773555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb40030840 con 0x7feb5006c0f0 2024-04-15T15:15:17.548 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.545+0000 7feb550c3640 1 -- 172.21.15.46:0/275773555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb40030c10 con 0x7feb5006c0f0 2024-04-15T15:15:17.548 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.545+0000 7feb58329640 1 -- 172.21.15.46:0/275773555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb5006c0f0 msgr2=0x7feb500b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.545+0000 7feb58329640 1 --2- 172.21.15.46:0/275773555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb5006c0f0 0x7feb500b06e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7feb40009870 tx=0x7feb4002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.546+0000 7feb58329640 1 -- 172.21.15.46:0/275773555 shutdown_connections 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.546+0000 7feb58329640 1 --2- 172.21.15.46:0/275773555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb5006c0f0 0x7feb500b06e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.546+0000 7feb58329640 1 --2- 172.21.15.46:0/275773555 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb501459c0 0x7feb500731f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.546+0000 7feb58329640 1 -- 172.21.15.46:0/275773555 >> 172.21.15.46:0/275773555 conn(0x7feb500b6a20 msgr2=0x7feb500b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.546+0000 7feb58329640 1 -- 172.21.15.46:0/275773555 shutdown_connections 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.546+0000 7feb58329640 1 -- 172.21.15.46:0/275773555 wait complete. 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.547+0000 7feb58329640 1 Processor -- start 2024-04-15T15:15:17.549 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.547+0000 7feb58329640 1 -- start start 2024-04-15T15:15:17.551 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.547+0000 7feb58329640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb5006c0f0 0x7feb500d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 0x7feb500cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb50145f60 0x7feb500cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb500b3920 con 0x7feb501459c0 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feb500b3c90 con 0x7feb50145f60 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7feb500b0c20 con 0x7feb5006c0f0 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 0x7feb500cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 0x7feb500cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37234/0 (socket says 172.21.15.46:37234) 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb560c5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb5006c0f0 0x7feb500d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 -- 172.21.15.46:0/3421844257 learned_addr learned my addr 172.21.15.46:0/3421844257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb568c6640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb50145f60 0x7feb500cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 -- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb5006c0f0 msgr2=0x7feb500d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb5006c0f0 0x7feb500d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 -- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb50145f60 msgr2=0x7feb500cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb50145f60 0x7feb500cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 -- 172.21.15.46:0/3421844257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb501f7b70 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb558c4640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 0x7feb500cd640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c730 tx=0x7feb4400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb44016020 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb44006d50 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb44003bb0 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb5014d1f0 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.548+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb500b5700 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.549+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feb4400abe0 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.549+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7feb44003d50 con 0x7feb501459c0 2024-04-15T15:15:17.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.549+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb5014d3c0 con 0x7feb501459c0 2024-04-15T15:15:17.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.551+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7feb44020020 con 0x7feb501459c0 2024-04-15T15:15:17.615 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.614+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb500d1140 con 0x7feb501459c0 2024-04-15T15:15:17.616 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.614+0000 7feb4effd640 1 -- 172.21.15.46:0/3421844257 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7feb44012420 con 0x7feb501459c0 2024-04-15T15:15:17.616 INFO:teuthology.misc.health.smithi046.stdout: 2024-04-15T15:15:17.616 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:16.367003+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:17.617 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 msgr2=0x7feb500cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.617 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 0x7feb500cd640 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c730 tx=0x7feb4400cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.617 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 shutdown_connections 2024-04-15T15:15:17.617 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb50145f60 0x7feb500cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.618 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb501459c0 0x7feb500cd640 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.618 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 --2- 172.21.15.46:0/3421844257 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb5006c0f0 0x7feb500d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.618 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 >> 172.21.15.46:0/3421844257 conn(0x7feb500b6a20 msgr2=0x7feb50141790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.618 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 shutdown_connections 2024-04-15T15:15:17.618 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:17.615+0000 7feb58329640 1 -- 172.21.15.46:0/3421844257 wait complete. 2024-04-15T15:15:17.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-15T15:15:17.900 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:17.898+0000 7f4a1243e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:15:17.972 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:17.970+0000 7fda0e23b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:15:18.177 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:18.175+0000 7f4a1243e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:15:18.228 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:18.226+0000 7f4a1243e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:15:18.263 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:18.261+0000 7fda0e23b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:15:18.316 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:18.314+0000 7fda0e23b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:15:18.342 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:18.340+0000 7f4a1243e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:15:18.440 INFO:tasks.ceph.mgr.y.smithi046.stderr:2024-04-15T15:15:18.438+0000 7fda0e23b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:15:19.358 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:19.356+0000 7f49b30c2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-15T15:15:19.358 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:15:19.357+0000 7f49b30c2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-15T15:15:20.464 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-04-15T15:15:20.463+0000 7fd895ffa640 -1 osd.2 0 log_to_monitors true 2024-04-15T15:15:20.465 INFO:tasks.ceph.osd.4.smithi078.stderr:2024-04-15T15:15:20.463+0000 7f8d4c8d3640 -1 osd.4 0 log_to_monitors true 2024-04-15T15:15:20.512 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-04-15T15:15:20.510+0000 7f33890a4640 -1 osd.3 0 log_to_monitors true 2024-04-15T15:15:20.517 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-04-15T15:15:20.515+0000 7f46c7437640 -1 osd.0 0 log_to_monitors true 2024-04-15T15:15:20.530 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-04-15T15:15:20.528+0000 7fe4bdb9e640 -1 osd.1 0 log_to_monitors true 2024-04-15T15:15:20.533 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-04-15T15:15:20.532+0000 7fa2ee391640 -1 osd.6 0 log_to_monitors true 2024-04-15T15:15:20.540 INFO:tasks.ceph.osd.7.smithi078.stderr:2024-04-15T15:15:20.538+0000 7f7763e1b640 -1 osd.7 0 log_to_monitors true 2024-04-15T15:15:20.547 INFO:tasks.ceph.osd.5.smithi078.stderr:2024-04-15T15:15:20.546+0000 7f29b3591640 -1 osd.5 0 log_to_monitors true 2024-04-15T15:15:23.740 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-04-15T15:15:23.737+0000 7fd891f90640 -1 osd.2 0 waiting for initial osdmap 2024-04-15T15:15:23.749 INFO:tasks.ceph.osd.2.smithi046.stderr:2024-04-15T15:15:23.748+0000 7fd889db0640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:23.789 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-04-15T15:15:23.787+0000 7fe4b9b34640 -1 osd.1 0 waiting for initial osdmap 2024-04-15T15:15:23.803 INFO:tasks.ceph.osd.1.smithi046.stderr:2024-04-15T15:15:23.802+0000 7fe4b1954640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:23.807 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-04-15T15:15:23.805+0000 7f46c33cd640 -1 osd.0 0 waiting for initial osdmap 2024-04-15T15:15:23.825 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-04-15T15:15:23.823+0000 7f338503a640 -1 osd.3 0 waiting for initial osdmap 2024-04-15T15:15:23.825 INFO:tasks.ceph.osd.0.smithi046.stderr:2024-04-15T15:15:23.824+0000 7f46bb1ed640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:23.838 INFO:tasks.ceph.osd.3.smithi046.stderr:2024-04-15T15:15:23.837+0000 7f337ce5a640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:24.434 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:24.466 INFO:tasks.ceph.osd.7.smithi078.stderr:2024-04-15T15:15:24.464+0000 7f775f6ed640 -1 osd.7 0 waiting for initial osdmap 2024-04-15T15:15:24.482 INFO:tasks.ceph.osd.7.smithi078.stderr:2024-04-15T15:15:24.480+0000 7f775750d640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:24.482 INFO:tasks.ceph.osd.5.smithi078.stderr:2024-04-15T15:15:24.481+0000 7f29af527640 -1 osd.5 0 waiting for initial osdmap 2024-04-15T15:15:24.505 INFO:tasks.ceph.osd.5.smithi078.stderr:2024-04-15T15:15:24.504+0000 7f29a7347640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:24.510 INFO:tasks.ceph.osd.4.smithi078.stderr:2024-04-15T15:15:24.508+0000 7f8d48869640 -1 osd.4 0 waiting for initial osdmap 2024-04-15T15:15:24.521 INFO:tasks.ceph.osd.4.smithi078.stderr:2024-04-15T15:15:24.520+0000 7f8d40689640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:24.534 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-04-15T15:15:24.532+0000 7fa2ea327640 -1 osd.6 0 waiting for initial osdmap 2024-04-15T15:15:24.542 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.540+0000 7f6a9b19d640 1 Processor -- start 2024-04-15T15:15:24.542 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.540+0000 7f6a9b19d640 1 -- start start 2024-04-15T15:15:24.549 INFO:tasks.ceph.osd.6.smithi078.stderr:2024-04-15T15:15:24.544+0000 7fa2e2147640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:15:24.551 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9b19d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6a9406d500 0x7f6a9406d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.551 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9b19d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a940d1fe0 0x7f6a9409b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.551 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9b19d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a940d1540 con 0x7f6a9415eb40 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9b19d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6a940d1b00 con 0x7f6a940d1fe0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9b19d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f6a940ccf40 con 0x7f6a9406d500 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9973a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a940d1fe0 0x7f6a9409b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9973a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6a940d1fe0 0x7f6a9409b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58376/0 (socket says 172.21.15.46:58376) 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a9973a640 1 -- 172.21.15.46:0/3358084036 learned_addr learned my addr 172.21.15.46:0/3358084036 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.541+0000 7f6a98f39640 1 --2- 172.21.15.46:0/3358084036 >> v2:172.21.15.78:3300/0 conn(0x7f6a9406d500 0x7f6a9406d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.542+0000 7f6a9973a640 1 -- 172.21.15.46:0/3358084036 >> v2:172.21.15.78:3300/0 conn(0x7f6a9406d500 msgr2=0x7f6a9406d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.542+0000 7f6a9973a640 1 --2- 172.21.15.46:0/3358084036 >> v2:172.21.15.78:3300/0 conn(0x7f6a9406d500 0x7f6a9406d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.542+0000 7f6a9973a640 1 -- 172.21.15.46:0/3358084036 >> v1:172.21.15.78:6789/0 conn(0x7f6a9415eb40 legacy=0x7f6a9409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.542+0000 7f6a9973a640 1 -- 172.21.15.46:0/3358084036 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a940d1820 con 0x7f6a940d1fe0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.542+0000 7f6a9973a640 1 --2- 172.21.15.46:0/3358084036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a9409b150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6a9406f6b0 tx=0x7f6a8c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8ccbc1f8f8f207 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a937fe640 1 -- 172.21.15.46:0/3358084036 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c02e070 con 0x7f6a940d1fe0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a937fe640 1 -- 172.21.15.46:0/3358084036 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a8c037070 con 0x7f6a940d1fe0 2024-04-15T15:15:24.552 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a937fe640 1 -- 172.21.15.46:0/3358084036 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c031a00 con 0x7f6a940d1fe0 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3358084036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 msgr2=0x7f6a9409b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3358084036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a9409b150 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6a9406f6b0 tx=0x7f6a8c02b880 comp rx=0 tx=0).stop 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3358084036 shutdown_connections 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3358084036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a9409b150 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3358084036 >> v2:172.21.15.78:3300/0 conn(0x7f6a9406d500 0x7f6a9406d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3358084036 >> 172.21.15.46:0/3358084036 conn(0x7f6a940b3f40 msgr2=0x7f6a940b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3358084036 shutdown_connections 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.543+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3358084036 wait complete. 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.544+0000 7f6a9b19d640 1 Processor -- start 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 -- start start 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6a9406d500 0x7f6a940c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a941f4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940cbf20 0x7f6a940ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.553 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6a940d1160 con 0x7f6a940d1fe0 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9406de20 con 0x7f6a940cbf20 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.545+0000 7f6a9b19d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6a940d12e0 con 0x7f6a9406d500 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a941f4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a941f4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58386/0 (socket says 172.21.15.46:58386) 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 -- 172.21.15.46:0/3630407670 learned_addr learned my addr 172.21.15.46:0/3630407670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a98f39640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6a9406d500 0x7f6a940c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a9973a640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940cbf20 0x7f6a940ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 -- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6a9406d500 msgr2=0x7f6a940c6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6a9406d500 0x7f6a940c6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 -- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940cbf20 msgr2=0x7f6a940ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940cbf20 0x7f6a940ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 -- 172.21.15.46:0/3630407670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a941f5380 con 0x7f6a940d1fe0 2024-04-15T15:15:24.554 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a9973a640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940cbf20 0x7f6a940ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a93fff640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a941f4e40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c3c0 tx=0x7f6a7c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c0052d0 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7c005490 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a9415df50 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c007260 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.546+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a940b6570 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.547+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a94075b20 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.548+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f6a7c007400 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.548+0000 7f6a917fa640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6a7406c890 0x7f6a7406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.548+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3631+0+0 (secure 0 0 0) 0x7f6a7c08a640 con 0x7f6a940d1fe0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.548+0000 7f6a98f39640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6a7406c890 0x7f6a7406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.555 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.549+0000 7f6a98f39640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6a7406c890 0x7f6a7406ed50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a840074a0 tx=0x7f6a8400b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.556 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.553+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6a941f5380 con 0x7f6a940d1fe0 2024-04-15T15:15:24.742 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.740+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a940b1080 con 0x7f6a940d1fe0 2024-04-15T15:15:24.742 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.741+0000 7f6a917fa640 1 -- 172.21.15.46:0/3630407670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+9439 (secure 0 0 0) 0x7f6a7c059400 con 0x7f6a940d1fe0 2024-04-15T15:15:24.742 INFO:teuthology.misc.health.smithi046.stdout: 2024-04-15T15:15:24.743 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":13,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:24.492572+0000","last_up_change":"2024-04-15T15:15:24.492572+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6a7406c890 msgr2=0x7f6a7406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6a7406c890 0x7f6a7406ed50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6a840074a0 tx=0x7f6a8400b040 comp rx=0 tx=0).stop 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 msgr2=0x7f6a941f4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a941f4e40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c3c0 tx=0x7f6a7c00c880 comp rx=0 tx=0).stop 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 shutdown_connections 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6a7406c890 0x7f6a7406ed50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a940cbf20 0x7f6a940ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6a940d1fe0 0x7f6a941f4e40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 --2- 172.21.15.46:0/3630407670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6a9406d500 0x7f6a940c6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 >> 172.21.15.46:0/3630407670 conn(0x7f6a940b3f40 msgr2=0x7f6a94071270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 shutdown_connections 2024-04-15T15:15:24.745 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:24.743+0000 7f6a9b19d640 1 -- 172.21.15.46:0/3630407670 wait complete. 2024-04-15T15:15:24.754 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-04-15T15:15:31.561 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:31.665 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.663+0000 7fe4ad152640 1 Processor -- start 2024-04-15T15:15:31.665 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.663+0000 7fe4ad152640 1 -- start start 2024-04-15T15:15:31.666 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4ad152640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe4a8161e10 0x7fe4a80b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.666 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4ad152640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80d4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.666 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4ad152640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4a80d0fe0 0x7fe4a80d1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4ad152640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe4a80d23f0 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4ad152640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe4a80cfed0 con 0x7fe4a8161e10 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4ad152640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a80d01b0 con 0x7fe4a80b5620 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4a80d0fe0 0x7fe4a80d1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe4a80d0fe0 0x7fe4a80d1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49612/0 (socket says 172.21.15.46:49612) 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 -- 172.21.15.46:0/3016686766 learned_addr learned my addr 172.21.15.46:0/3016686766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe49e59c640 1 --2- 172.21.15.46:0/3016686766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80d4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a6d9d640 1 --2- 172.21.15.46:0/3016686766 >> v2:172.21.15.78:3300/0 conn(0x7fe4a8161e10 0x7fe4a80b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 -- 172.21.15.46:0/3016686766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 msgr2=0x7fe4a80d4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 --2- 172.21.15.46:0/3016686766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80d4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 -- 172.21.15.46:0/3016686766 >> v2:172.21.15.78:3300/0 conn(0x7fe4a8161e10 msgr2=0x7fe4a80b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 --2- 172.21.15.46:0/3016686766 >> v2:172.21.15.78:3300/0 conn(0x7fe4a8161e10 0x7fe4a80b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.664+0000 7fe4a759e640 1 -- 172.21.15.46:0/3016686766 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a80d0490 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.667 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.665+0000 7fe4a759e640 1 --2- 172.21.15.46:0/3016686766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a80d1470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80d4570 tx=0x7fe4a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eea1277371eeb98 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.665+0000 7fe4a659c640 1 -- 172.21.15.46:0/3016686766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a002c9e0 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.665+0000 7fe4a659c640 1 -- 172.21.15.46:0/3016686766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a002cb80 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.665+0000 7fe4a659c640 1 -- 172.21.15.46:0/3016686766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00343f0 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 -- 172.21.15.46:0/3016686766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 msgr2=0x7fe4a80d1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 --2- 172.21.15.46:0/3016686766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a80d1470 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4a80d4570 tx=0x7fe4a002be40 comp rx=0 tx=0).stop 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 -- 172.21.15.46:0/3016686766 shutdown_connections 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 --2- 172.21.15.46:0/3016686766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a80d1470 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 --2- 172.21.15.46:0/3016686766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80d4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.669 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 --2- 172.21.15.46:0/3016686766 >> v2:172.21.15.78:3300/0 conn(0x7fe4a8161e10 0x7fe4a80b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.670 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 -- 172.21.15.46:0/3016686766 >> 172.21.15.46:0/3016686766 conn(0x7fe4a808bfd0 msgr2=0x7fe4a80d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.670 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 -- 172.21.15.46:0/3016686766 shutdown_connections 2024-04-15T15:15:31.670 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.666+0000 7fe4ad152640 1 -- 172.21.15.46:0/3016686766 wait complete. 2024-04-15T15:15:31.670 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.667+0000 7fe4ad152640 1 Processor -- start 2024-04-15T15:15:31.670 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.667+0000 7fe4ad152640 1 -- start start 2024-04-15T15:15:31.671 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4ad152640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4ad152640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a8148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4ad152640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8161e10 0x7fe4a8155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4ad152640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8162370 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4ad152640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a81621f0 con 0x7fe4a8161e10 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4ad152640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe4a80d2880 con 0x7fe4a80b5620 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a8148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a8148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49618/0 (socket says 172.21.15.46:49618) 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 -- 172.21.15.46:0/2735276056 learned_addr learned my addr 172.21.15.46:0/2735276056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe4a6d9d640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 -- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 msgr2=0x7fe4a80c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80c7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 -- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8161e10 msgr2=0x7fe4a8155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.672 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8161e10 0x7fe4a8155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.668+0000 7fe49e59c640 1 -- 172.21.15.46:0/2735276056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a81561d0 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.669+0000 7fe49e59c640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a8148960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe49000e890 tx=0x7fe49000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.669+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe490017070 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.669+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe49000c940 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.669+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a8148ea0 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.669+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe49000cd10 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.669+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a80c8680 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.673 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.670+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a8075b20 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.674 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.670+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe49001a070 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.674 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.671+0000 7fe49ffff640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe48806c8f0 0x7fe48806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.674 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.671+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe49008b830 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.674 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.671+0000 7fe4a6d9d640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe48806c8f0 0x7fe48806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.674 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.672+0000 7fe4a6d9d640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe48806c8f0 0x7fe48806edb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe498009970 tx=0x7fe498008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.677 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.676+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe490059f30 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.863 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.861+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4a80d1d00 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.864 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.862+0000 7fe49ffff640 1 -- 172.21.15.46:0/2735276056 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12836 (secure 0 0 0) 0x7fe490059830 con 0x7fe4a80d0fe0 2024-04-15T15:15:31.864 INFO:teuthology.misc.health.smithi046.stdout: 2024-04-15T15:15:31.864 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":17,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:28.519563+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:31.866 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe48806c8f0 msgr2=0x7fe48806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.866 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe48806c8f0 0x7fe48806edb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe498009970 tx=0x7fe498008040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.866 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 msgr2=0x7fe4a8148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.866 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a8148960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe49000e890 tx=0x7fe49000ed50 comp rx=0 tx=0).stop 2024-04-15T15:15:31.866 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 shutdown_connections 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe48806c8f0 0x7fe48806edb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8161e10 0x7fe4a8155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4a80d0fe0 0x7fe4a8148960 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 --2- 172.21.15.46:0/2735276056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe4a80b5620 0x7fe4a80c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 >> 172.21.15.46:0/2735276056 conn(0x7fe4a808bfd0 msgr2=0x7fe4a815d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 shutdown_connections 2024-04-15T15:15:31.867 INFO:teuthology.misc.health.smithi046.stderr:2024-04-15T15:15:31.864+0000 7fe4ad152640 1 -- 172.21.15.46:0/2735276056 wait complete. 2024-04-15T15:15:31.877 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-15T15:15:31.877 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-15T15:15:31.878 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:15:31.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:15:31.880 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:15:31.880 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:15:31.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:15:31.883 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:15:31.883 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:15:31.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:15:31.886 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:15:31.886 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:15:31.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:15:31.889 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:15:31.889 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-15T15:15:31.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:31.959 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-04-15T15:15:31.960 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-04-15T15:15:31.976 INFO:tasks.ceph.mds.b.smithi078.stdout:starting mds.b at 2024-04-15T15:15:31.976 INFO:tasks.ceph.mds.d.smithi078.stdout:starting mds.d at 2024-04-15T15:15:31.976 INFO:tasks.ceph.mds.b.smithi078.stderr:2024-04-15T15:15:31.975+0000 7fabe762f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-15T15:15:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.997+0000 7fe33977c640 1 Processor -- start 2024-04-15T15:15:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.997+0000 7fe33977c640 1 -- start start 2024-04-15T15:15:32.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe33977c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe334162cd0 0x7fe33406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe33977c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3340cde10 0x7fe3340c4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe33977c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3340c4aa0 0x7fe33406f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe33977c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe3340d1f40 con 0x7fe3340cde10 2024-04-15T15:15:32.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe33977c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe3340ce360 con 0x7fe3340c4aa0 2024-04-15T15:15:32.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe33977c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe3340ce640 con 0x7fe334162cd0 2024-04-15T15:15:32.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3340cde10 0x7fe3340c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3340cde10 0x7fe3340c4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49662/0 (socket says 172.21.15.46:49662) 2024-04-15T15:15:32.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 -- 172.21.15.46:0/4242364965 learned_addr learned my addr 172.21.15.46:0/4242364965 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:32.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe332ffd640 1 --2- 172.21.15.46:0/4242364965 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe334162cd0 0x7fe33406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3337fe640 1 --2- 172.21.15.46:0/4242364965 >> v2:172.21.15.78:3300/0 conn(0x7fe3340c4aa0 0x7fe33406f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 -- 172.21.15.46:0/4242364965 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe334162cd0 msgr2=0x7fe33406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 --2- 172.21.15.46:0/4242364965 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe334162cd0 0x7fe33406c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 -- 172.21.15.46:0/4242364965 >> v2:172.21.15.78:3300/0 conn(0x7fe3340c4aa0 msgr2=0x7fe33406f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 --2- 172.21.15.46:0/4242364965 >> v2:172.21.15.78:3300/0 conn(0x7fe3340c4aa0 0x7fe33406f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.998+0000 7fe3327fc640 1 -- 172.21.15.46:0/4242364965 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3340ce920 con 0x7fe3340cde10 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.999+0000 7fe3327fc640 1 --2- 172.21.15.46:0/4242364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3340cde10 0x7fe3340c4560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe33406f780 tx=0x7fe31c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e4a30167834a48 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.999+0000 7fe331ffb640 1 -- 172.21.15.46:0/4242364965 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c035020 con 0x7fe3340cde10 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.999+0000 7fe331ffb640 1 -- 172.21.15.46:0/4242364965 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c031870 con 0x7fe3340cde10 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:31.999+0000 7fe331ffb640 1 -- 172.21.15.46:0/4242364965 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31c031c40 con 0x7fe3340cde10 2024-04-15T15:15:32.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 -- 172.21.15.46:0/4242364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3340cde10 msgr2=0x7fe3340c4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 --2- 172.21.15.46:0/4242364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3340cde10 0x7fe3340c4560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe33406f780 tx=0x7fe31c02b910 comp rx=0 tx=0).stop 2024-04-15T15:15:32.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 -- 172.21.15.46:0/4242364965 shutdown_connections 2024-04-15T15:15:32.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 --2- 172.21.15.46:0/4242364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3340cde10 0x7fe3340c4560 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 --2- 172.21.15.46:0/4242364965 >> v2:172.21.15.78:3300/0 conn(0x7fe3340c4aa0 0x7fe33406f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 --2- 172.21.15.46:0/4242364965 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe334162cd0 0x7fe33406c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 -- 172.21.15.46:0/4242364965 >> 172.21.15.46:0/4242364965 conn(0x7fe33409a9d0 msgr2=0x7fe3340d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 -- 172.21.15.46:0/4242364965 shutdown_connections 2024-04-15T15:15:32.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.000+0000 7fe33977c640 1 -- 172.21.15.46:0/4242364965 wait complete. 2024-04-15T15:15:32.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.001+0000 7fe33977c640 1 Processor -- start 2024-04-15T15:15:32.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.001+0000 7fe33977c640 1 -- start start 2024-04-15T15:15:32.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe33977c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3340c4aa0 0x7fe334164b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe33977c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3340cde10 0x7fe334157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe33977c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 0x7fe334163230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe33977c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341630b0 con 0x7fe334162cd0 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe33977c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3340ceee0 con 0x7fe3340cde10 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe33977c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe33406fd30 con 0x7fe3340c4aa0 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 0x7fe334163230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe332ffd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3340c4aa0 0x7fe334164b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe332ffd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3340c4aa0 0x7fe334164b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:60264/0 (socket says 172.21.15.46:60264) 2024-04-15T15:15:32.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3327fc640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3340cde10 0x7fe334157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe332ffd640 1 -- 172.21.15.46:0/435731484 learned_addr learned my addr 172.21.15.46:0/435731484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:32.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 0x7fe334163230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49678/0 (socket says 172.21.15.46:49678) 2024-04-15T15:15:32.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 -- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3340c4aa0 msgr2=0x7fe334164b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3340c4aa0 0x7fe334164b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 -- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3340cde10 msgr2=0x7fe334157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3340cde10 0x7fe334157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 -- 172.21.15.46:0/435731484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe334165060 con 0x7fe334162cd0 2024-04-15T15:15:32.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe3337fe640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 0x7fe334163230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00ef00 tx=0x7fe32c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c018020 con 0x7fe334162cd0 2024-04-15T15:15:32.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.002+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32c014400 con 0x7fe334162cd0 2024-04-15T15:15:32.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.003+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3341582f0 con 0x7fe334162cd0 2024-04-15T15:15:32.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.003+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c009e60 con 0x7fe334162cd0 2024-04-15T15:15:32.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.003+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe334163d60 con 0x7fe334162cd0 2024-04-15T15:15:32.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.003+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3340690f0 con 0x7fe334162cd0 2024-04-15T15:15:32.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.004+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe32c005d10 con 0x7fe334162cd0 2024-04-15T15:15:32.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.004+0000 7fe32b7fe640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe31406c920 0x7fe31406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.004+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fe32c08bf50 con 0x7fe334162cd0 2024-04-15T15:15:32.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.005+0000 7fe332ffd640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe31406c920 0x7fe31406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.005+0000 7fe332ffd640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe31406c920 0x7fe31406ede0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe324009b60 tx=0x7fe324009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.009+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe32c022030 con 0x7fe334162cd0 2024-04-15T15:15:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.198+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3340c7ef0 con 0x7fe334162cd0 2024-04-15T15:15:32.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.199+0000 7fe32b7fe640 1 -- 172.21.15.46:0/435731484 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12836 (secure 0 0 0) 0x7fe32c05a110 con 0x7fe334162cd0 2024-04-15T15:15:32.204 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:32.204 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:28.519563+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe31406c920 msgr2=0x7fe31406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe31406c920 0x7fe31406ede0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe324009b60 tx=0x7fe324009190 comp rx=0 tx=0).stop 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 msgr2=0x7fe334163230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 0x7fe334163230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00ef00 tx=0x7fe32c00b480 comp rx=0 tx=0).stop 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 shutdown_connections 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe31406c920 0x7fe31406ede0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe334162cd0 0x7fe334163230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3340cde10 0x7fe334157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 --2- 172.21.15.46:0/435731484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3340c4aa0 0x7fe334164b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 >> 172.21.15.46:0/435731484 conn(0x7fe33409a9d0 msgr2=0x7fe3340700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 shutdown_connections 2024-04-15T15:15:32.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.201+0000 7fe33977c640 1 -- 172.21.15.46:0/435731484 wait complete. 2024-04-15T15:15:32.213 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:15:32.214 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.319+0000 7f3316f09640 1 Processor -- start 2024-04-15T15:15:32.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.319+0000 7f3316f09640 1 -- start start 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3316f09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f331015e700 0x7f33100b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3316f09640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33100d2360 0x7f33100d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3316f09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33100b47a0 con 0x7f331015eae0 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3316f09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f33100b53b0 con 0x7f331015e700 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3316f09640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f33100b4a80 con 0x7f33100d2360 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3314ca5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f331015e700 0x7f33100b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f33154a6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f331015eae0 0x7f33100cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55454/0 (socket says 172.21.15.46:55454) 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f33154a6640 1 -- 172.21.15.46:0/2927668184 learned_addr learned my addr 172.21.15.46:0/2927668184 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3314ca5640 1 --2- 172.21.15.46:0/2927668184 >> v2:172.21.15.46:3300/0 conn(0x7f331015e700 0x7f33100b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49694/0 (socket says 172.21.15.46:49694) 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f330ffff640 1 --2- 172.21.15.46:0/2927668184 >> v2:172.21.15.78:3300/0 conn(0x7f33100d2360 0x7f33100d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3314ca5640 1 -- 172.21.15.46:0/2927668184 >> v2:172.21.15.78:3300/0 conn(0x7f33100d2360 msgr2=0x7f33100d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3314ca5640 1 --2- 172.21.15.46:0/2927668184 >> v2:172.21.15.78:3300/0 conn(0x7f33100d2360 0x7f33100d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3314ca5640 1 -- 172.21.15.46:0/2927668184 >> v1:172.21.15.46:6789/0 conn(0x7f331015eae0 legacy=0x7f33100cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f3314ca5640 1 -- 172.21.15.46:0/2927668184 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33100b4d60 con 0x7f331015e700 2024-04-15T15:15:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.320+0000 7f330f7fe640 1 -- 172.21.15.46:0/2927668184 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336259466 0 0) 0x7f33100b47a0 con 0x7f331015eae0 2024-04-15T15:15:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.321+0000 7f3314ca5640 1 --2- 172.21.15.46:0/2927668184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015e700 0x7f33100b5840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f331008c7b0 tx=0x7f330402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80ceca6b0d95bca server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.321+0000 7f330f7fe640 1 -- 172.21.15.46:0/2927668184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304035020 con 0x7f331015e700 2024-04-15T15:15:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.321+0000 7f330f7fe640 1 -- 172.21.15.46:0/2927668184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3304031830 con 0x7f331015e700 2024-04-15T15:15:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.321+0000 7f330f7fe640 1 -- 172.21.15.46:0/2927668184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304031c00 con 0x7f331015e700 2024-04-15T15:15:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 -- 172.21.15.46:0/2927668184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015e700 msgr2=0x7f33100b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 --2- 172.21.15.46:0/2927668184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015e700 0x7f33100b5840 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f331008c7b0 tx=0x7f330402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 -- 172.21.15.46:0/2927668184 shutdown_connections 2024-04-15T15:15:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 --2- 172.21.15.46:0/2927668184 >> v2:172.21.15.78:3300/0 conn(0x7f33100d2360 0x7f33100d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 --2- 172.21.15.46:0/2927668184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015e700 0x7f33100b5840 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 -- 172.21.15.46:0/2927668184 >> 172.21.15.46:0/2927668184 conn(0x7f331008bf60 msgr2=0x7f33100d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 -- 172.21.15.46:0/2927668184 shutdown_connections 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.322+0000 7f3316f09640 1 -- 172.21.15.46:0/2927668184 wait complete. 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.323+0000 7f3316f09640 1 Processor -- start 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.323+0000 7f3316f09640 1 -- start start 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3316f09640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100d2360 0x7f33101f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3316f09640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f331015e700 0x7f33100b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3316f09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 0x7f33100b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3316f09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33100b5d80 con 0x7f331015eae0 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3316f09640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33100b4450 con 0x7f33100d2360 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3316f09640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f33100d1d40 con 0x7f331015e700 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 0x7f33100b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 0x7f33100b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49700/0 (socket says 172.21.15.46:49700) 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 -- 172.21.15.46:0/2495871152 learned_addr learned my addr 172.21.15.46:0/2495871152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f3314ca5640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100d2360 0x7f33101f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f330ffff640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f331015e700 0x7f33100b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 -- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f331015e700 msgr2=0x7f33100b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f331015e700 0x7f33100b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 -- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100d2360 msgr2=0x7f33101f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100d2360 0x7f33101f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 -- 172.21.15.46:0/2495871152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33100b4230 con 0x7f331015eae0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.324+0000 7f33154a6640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 0x7f33100b32d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f330800c700 tx=0x7f330800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.325+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308018070 con 0x7f331015eae0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.325+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f330800ce10 con 0x7f331015eae0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.325+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3310148990 con 0x7f331015eae0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.325+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330800a880 con 0x7f331015eae0 2024-04-15T15:15:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.325+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3310148f10 con 0x7f331015eae0 2024-04-15T15:15:32.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.326+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33100ca360 con 0x7f331015eae0 2024-04-15T15:15:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.326+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f330801b070 con 0x7f331015eae0 2024-04-15T15:15:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.327+0000 7f330d7fa640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f406c8f0 0x7f32f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.327+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f330808ac70 con 0x7f331015eae0 2024-04-15T15:15:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.327+0000 7f3314ca5640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f406c8f0 0x7f32f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.327+0000 7f3314ca5640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f406c8f0 0x7f32f406edb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3304036000 tx=0x7f3304008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.331+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33100ca360 con 0x7f331015eae0 2024-04-15T15:15:32.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.514+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f33100689e0 con 0x7f331015eae0 2024-04-15T15:15:32.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.514+0000 7f330d7fa640 1 -- 172.21.15.46:0/2495871152 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f3308058da0 con 0x7f331015eae0 2024-04-15T15:15:32.516 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f406c8f0 msgr2=0x7f32f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f406c8f0 0x7f32f406edb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3304036000 tx=0x7f3304008030 comp rx=0 tx=0).stop 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 msgr2=0x7f33100b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 0x7f33100b32d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f330800c700 tx=0x7f330800cbc0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 shutdown_connections 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f406c8f0 0x7f32f406edb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f331015eae0 0x7f33100b32d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f331015e700 0x7f33100b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 --2- 172.21.15.46:0/2495871152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100d2360 0x7f33101f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 >> 172.21.15.46:0/2495871152 conn(0x7f331008bf60 msgr2=0x7f33100d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 shutdown_connections 2024-04-15T15:15:32.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.517+0000 7f3316f09640 1 -- 172.21.15.46:0/2495871152 wait complete. 2024-04-15T15:15:32.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.632+0000 7f0b092e0640 1 Processor -- start 2024-04-15T15:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.632+0000 7f0b092e0640 1 -- start start 2024-04-15T15:15:32.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b092e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b0415e790 0x7f0b040d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b092e0640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040cfd20 0x7f0b040d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b092e0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b040d2480 con 0x7f0b040d0f00 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b092e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0b040d2db0 con 0x7f0b0415e790 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b092e0640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b040d2760 con 0x7f0b040cfd20 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b0415e790 0x7f0b040d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b0415e790 0x7f0b040d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49710/0 (socket says 172.21.15.46:49710) 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 -- 172.21.15.46:0/4250380731 learned_addr learned my addr 172.21.15.46:0/4250380731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b027fc640 1 --2- 172.21.15.46:0/4250380731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040cfd20 0x7f0b040d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 -- 172.21.15.46:0/4250380731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040cfd20 msgr2=0x7f0b040d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 --2- 172.21.15.46:0/4250380731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040cfd20 0x7f0b040d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 -- 172.21.15.46:0/4250380731 >> v1:172.21.15.78:6789/0 conn(0x7f0b040d0f00 legacy=0x7f0b040b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.633+0000 7f0b02ffd640 1 -- 172.21.15.46:0/4250380731 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b040d2a40 con 0x7f0b0415e790 2024-04-15T15:15:32.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.634+0000 7f0b02ffd640 1 --2- 172.21.15.46:0/4250380731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040d3240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009870 tx=0x7f0af402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1e632f198a56ad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.634+0000 7f0b01ffb640 1 -- 172.21.15.46:0/4250380731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af4035020 con 0x7f0b0415e790 2024-04-15T15:15:32.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.634+0000 7f0b01ffb640 1 -- 172.21.15.46:0/4250380731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af4031640 con 0x7f0b0415e790 2024-04-15T15:15:32.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.634+0000 7f0b01ffb640 1 -- 172.21.15.46:0/4250380731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af402d040 con 0x7f0b0415e790 2024-04-15T15:15:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.634+0000 7f0b092e0640 1 -- 172.21.15.46:0/4250380731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 msgr2=0x7f0b040d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.634+0000 7f0b092e0640 1 --2- 172.21.15.46:0/4250380731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040d3240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009870 tx=0x7f0af402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.635+0000 7f0b092e0640 1 -- 172.21.15.46:0/4250380731 shutdown_connections 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.635+0000 7f0b092e0640 1 --2- 172.21.15.46:0/4250380731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040cfd20 0x7f0b040d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.635+0000 7f0b092e0640 1 --2- 172.21.15.46:0/4250380731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040d3240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.635+0000 7f0b092e0640 1 -- 172.21.15.46:0/4250380731 >> 172.21.15.46:0/4250380731 conn(0x7f0b0408bf60 msgr2=0x7f0b040d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.635+0000 7f0b092e0640 1 -- 172.21.15.46:0/4250380731 shutdown_connections 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.635+0000 7f0b092e0640 1 -- 172.21.15.46:0/4250380731 wait complete. 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.636+0000 7f0b092e0640 1 Processor -- start 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.636+0000 7f0b092e0640 1 -- start start 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b040cfd20 0x7f0b040cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040d0f00 0x7f0b040c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0415ecf0 con 0x7f0b0415e790 2024-04-15T15:15:32.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0415eb70 con 0x7f0b040cfd20 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b040d13b0 con 0x7f0b040d0f00 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49714/0 (socket says 172.21.15.46:49714) 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 -- 172.21.15.46:0/879563354 learned_addr learned my addr 172.21.15.46:0/879563354 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b02ffd640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b040cfd20 0x7f0b040cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b027fc640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040d0f00 0x7f0b040c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 -- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040d0f00 msgr2=0x7f0b040c7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040d0f00 0x7f0b040c7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 -- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b040cfd20 msgr2=0x7f0b040cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b040cfd20 0x7f0b040cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 -- 172.21.15.46:0/879563354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b040c8410 con 0x7f0b0415e790 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b037fe640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040c7380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c780 tx=0x7f0afc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc004b60 con 0x7f0b0415e790 2024-04-15T15:15:32.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0afc004d00 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b040c86f0 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00aa90 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.637+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b040c8cb0 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.638+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b040689e0 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.639+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0afc00ac30 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.639+0000 7f0aeb7fe640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0af006c8f0 0x7f0af006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.639+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0afc08b9a0 con 0x7f0b0415e790 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.640+0000 7f0b02ffd640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0af006c8f0 0x7f0af006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.640+0000 7f0b02ffd640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0af006c8f0 0x7f0af006edb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0af4036000 tx=0x7f0af403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.644+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0afc020070 con 0x7f0b0415e790 2024-04-15T15:15:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.869+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b040d1ce0 con 0x7f0b0415e790 2024-04-15T15:15:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.870+0000 7f0aeb7fe640 1 -- 172.21.15.46:0/879563354 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f0afc059ad0 con 0x7f0b0415e790 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0af006c8f0 msgr2=0x7f0af006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0af006c8f0 0x7f0af006edb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0af4036000 tx=0x7f0af403c040 comp rx=0 tx=0).stop 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 msgr2=0x7f0b040c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040c7380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0afc00c780 tx=0x7f0afc00cc40 comp rx=0 tx=0).stop 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 shutdown_connections 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0af006c8f0 0x7f0af006edb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b0415e790 0x7f0b040c7380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b040d0f00 0x7f0b040c7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 --2- 172.21.15.46:0/879563354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b040cfd20 0x7f0b040cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 >> 172.21.15.46:0/879563354 conn(0x7f0b0408bf60 msgr2=0x7f0b040caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 shutdown_connections 2024-04-15T15:15:32.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:32.872+0000 7f0b092e0640 1 -- 172.21.15.46:0/879563354 wait complete. 2024-04-15T15:15:34.886 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.001+0000 7fad5aef8640 1 Processor -- start 2024-04-15T15:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.001+0000 7fad5aef8640 1 -- start start 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad5aef8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad540d3240 0x7fad540d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad5aef8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fad540d1440 0x7fad540b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad5aef8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad540d2480 con 0x7fad5415e790 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad5aef8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fad540d2db0 con 0x7fad540d1440 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad5aef8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fad540d2760 con 0x7fad540d3240 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad4bfff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad540d3240 0x7fad540d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad59495640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fad540d1440 0x7fad540b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad58c94640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fad5415e790 0x7fad540cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:46620/0 (socket says 172.21.15.46:46620) 2024-04-15T15:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.002+0000 7fad58c94640 1 -- 172.21.15.46:0/785230427 learned_addr learned my addr 172.21.15.46:0/785230427 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.003+0000 7fad4bfff640 1 -- 172.21.15.46:0/785230427 >> v2:172.21.15.78:3300/0 conn(0x7fad540d1440 msgr2=0x7fad540b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.003+0000 7fad4bfff640 1 --2- 172.21.15.46:0/785230427 >> v2:172.21.15.78:3300/0 conn(0x7fad540d1440 0x7fad540b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.003+0000 7fad4bfff640 1 -- 172.21.15.46:0/785230427 >> v1:172.21.15.78:6789/0 conn(0x7fad5415e790 legacy=0x7fad540cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.003+0000 7fad4bfff640 1 -- 172.21.15.46:0/785230427 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad540d2a40 con 0x7fad540d3240 2024-04-15T15:15:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.003+0000 7fad4bfff640 1 --2- 172.21.15.46:0/785230427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad540d3240 0x7fad540d0f00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad440097a0 tx=0x7fad4402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6804950069c7d3a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad4b7fe640 1 -- 172.21.15.46:0/785230427 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44035020 con 0x7fad540d3240 2024-04-15T15:15:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad4b7fe640 1 -- 172.21.15.46:0/785230427 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad44031830 con 0x7fad540d3240 2024-04-15T15:15:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad4b7fe640 1 -- 172.21.15.46:0/785230427 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44031c00 con 0x7fad540d3240 2024-04-15T15:15:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 -- 172.21.15.46:0/785230427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad540d3240 msgr2=0x7fad540d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 --2- 172.21.15.46:0/785230427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad540d3240 0x7fad540d0f00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad440097a0 tx=0x7fad4402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 -- 172.21.15.46:0/785230427 shutdown_connections 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 --2- 172.21.15.46:0/785230427 >> v2:172.21.15.78:3300/0 conn(0x7fad540d1440 0x7fad540b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 --2- 172.21.15.46:0/785230427 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad540d3240 0x7fad540d0f00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 -- 172.21.15.46:0/785230427 >> 172.21.15.46:0/785230427 conn(0x7fad5408bf60 msgr2=0x7fad540d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 -- 172.21.15.46:0/785230427 shutdown_connections 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.004+0000 7fad5aef8640 1 -- 172.21.15.46:0/785230427 wait complete. 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.005+0000 7fad5aef8640 1 Processor -- start 2024-04-15T15:15:35.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.006+0000 7fad5aef8640 1 -- start start 2024-04-15T15:15:35.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.006+0000 7fad5aef8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 0x7fad54158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.006+0000 7fad5aef8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad540d3240 0x7fad540b9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.006+0000 7fad5aef8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad5415e790 0x7fad540b9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad5aef8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fad540d5180 con 0x7fad540d1440 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad5aef8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fad540d0430 con 0x7fad540d3240 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad5aef8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fad540d5300 con 0x7fad5415e790 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 0x7fad54158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 0x7fad54158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49730/0 (socket says 172.21.15.46:49730) 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 -- 172.21.15.46:0/2947384426 learned_addr learned my addr 172.21.15.46:0/2947384426 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad59495640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad5415e790 0x7fad540b9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad4bfff640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad540d3240 0x7fad540b9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 -- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad5415e790 msgr2=0x7fad540b9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad5415e790 0x7fad540b9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 -- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad540d3240 msgr2=0x7fad540b9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad540d3240 0x7fad540b9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 -- 172.21.15.46:0/2947384426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad541594c0 con 0x7fad540d1440 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad58c94640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 0x7fad54158f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fad5000c3c0 tx=0x7fad5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50017070 con 0x7fad540d1440 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad50005ce0 con 0x7fad540d1440 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad541ea880 con 0x7fad540d1440 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad500072f0 con 0x7fad540d1440 2024-04-15T15:15:35.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.007+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5415ab80 con 0x7fad540d1440 2024-04-15T15:15:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.008+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad540689e0 con 0x7fad540d1440 2024-04-15T15:15:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.009+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fad5001a070 con 0x7fad540d1440 2024-04-15T15:15:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.009+0000 7fad497fa640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad4006c920 0x7fad4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.009+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fad5008aa80 con 0x7fad540d1440 2024-04-15T15:15:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.010+0000 7fad4bfff640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad4006c920 0x7fad4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.010+0000 7fad4bfff640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad4006c920 0x7fad4006ede0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad44036000 tx=0x7fad44008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.014+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad50058bb0 con 0x7fad540d1440 2024-04-15T15:15:35.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.238+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad5415e240 con 0x7fad540d1440 2024-04-15T15:15:35.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.238+0000 7fad497fa640 1 -- 172.21.15.46:0/2947384426 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3861 (secure 0 0 0) 0x7fad5005ceb0 con 0x7fad540d1440 2024-04-15T15:15:35.240 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:35.240 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4284,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-15T15:15:35.241 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-04-15T15:15:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad4006c920 msgr2=0x7fad4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad4006c920 0x7fad4006ede0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fad44036000 tx=0x7fad44008030 comp rx=0 tx=0).stop 2024-04-15T15:15:35.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 msgr2=0x7fad54158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 0x7fad54158f80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fad5000c3c0 tx=0x7fad5000c880 comp rx=0 tx=0).stop 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 shutdown_connections 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad4006c920 0x7fad4006ede0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad5415e790 0x7fad540b9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad540d3240 0x7fad540b9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.240+0000 7fad5aef8640 1 --2- 172.21.15.46:0/2947384426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad540d1440 0x7fad54158f80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.241+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 >> 172.21.15.46:0/2947384426 conn(0x7fad5408bf60 msgr2=0x7fad540cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.241+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 shutdown_connections 2024-04-15T15:15:35.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.241+0000 7fad5aef8640 1 -- 172.21.15.46:0/2947384426 wait complete. 2024-04-15T15:15:35.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.356+0000 7f2abe61b640 1 Processor -- start 2024-04-15T15:15:35.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.356+0000 7f2abe61b640 1 -- start start 2024-04-15T15:15:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2abe61b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2abe61b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab80b47a0 con 0x7f2ab815e700 2024-04-15T15:15:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2abe61b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab80b4a80 con 0x7f2ab815eae0 2024-04-15T15:15:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2abe61b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab80b53b0 con 0x7f2ab80d2360 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ab815e700 0x7f2ab80b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55468/0 (socket says 172.21.15.46:55468) 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab7fff640 1 -- 172.21.15.46:0/1129684346 learned_addr learned my addr 172.21.15.46:0/1129684346 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab77fe640 1 --2- 172.21.15.46:0/1129684346 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918707957 0 0) 0x7f2ab80b47a0 con 0x7f2ab815e700 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa0003520 con 0x7f2ab815e700 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993956902 0 0) 0x7f2ab80b4a80 con 0x7f2ab815eae0 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ab80b47a0 con 0x7f2ab815eae0 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab77fe640 1 -- 172.21.15.46:0/1129684346 >> v1:172.21.15.78:6789/0 conn(0x7f2ab815eae0 legacy=0x7f2ab80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 612155529 0 0) 0x7f2aa0003520 con 0x7f2ab815e700 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab77fe640 1 -- 172.21.15.46:0/1129684346 >> v1:172.21.15.46:6789/0 conn(0x7f2ab815e700 legacy=0x7f2ab80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.357+0000 7f2ab77fe640 1 -- 172.21.15.46:0/1129684346 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab80b4d60 con 0x7f2ab80d2360 2024-04-15T15:15:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.358+0000 7f2ab77fe640 1 --2- 172.21.15.46:0/1129684346 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab80d2b60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80b5530 tx=0x7f2aac02b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=249f15857e0031f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.358+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac039070 con 0x7f2ab80d2360 2024-04-15T15:15:35.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.358+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac030070 con 0x7f2ab80d2360 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.358+0000 7f2ab6ffd640 1 -- 172.21.15.46:0/1129684346 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80017f0 con 0x7f2ab80d2360 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 -- 172.21.15.46:0/1129684346 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 msgr2=0x7f2ab80d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 --2- 172.21.15.46:0/1129684346 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab80d2b60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ab80b5530 tx=0x7f2aac02b7f0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 -- 172.21.15.46:0/1129684346 shutdown_connections 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 --2- 172.21.15.46:0/1129684346 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab80d2b60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 -- 172.21.15.46:0/1129684346 >> 172.21.15.46:0/1129684346 conn(0x7f2ab808bf60 msgr2=0x7f2ab80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 -- 172.21.15.46:0/1129684346 shutdown_connections 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.359+0000 7f2abe61b640 1 -- 172.21.15.46:0/1129684346 wait complete. 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.360+0000 7f2abe61b640 1 Processor -- start 2024-04-15T15:15:35.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.360+0000 7f2abe61b640 1 -- start start 2024-04-15T15:15:35.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abe61b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab8153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abe61b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ab815e700 0x7f2ab80cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abe61b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 0x7f2ab80ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abe61b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80d1d40 con 0x7f2ab815eae0 2024-04-15T15:15:35.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abe61b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80b4450 con 0x7f2ab815e700 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abe61b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab80d1ec0 con 0x7f2ab80d2360 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 0x7f2ab80ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 0x7f2ab80ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49740/0 (socket says 172.21.15.46:49740) 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 -- 172.21.15.46:0/3087152704 learned_addr learned my addr 172.21.15.46:0/3087152704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2ab7fff640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab8153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2ab77fe640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ab815e700 0x7f2ab80cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 -- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 msgr2=0x7f2ab8153870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab8153870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 -- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ab815e700 msgr2=0x7f2ab80cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ab815e700 0x7f2ab80cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.361+0000 7f2abcbb8640 1 -- 172.21.15.46:0/3087152704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab80c91d0 con 0x7f2ab815eae0 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.362+0000 7f2abcbb8640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 0x7f2ab80ca2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000c6e0 tx=0x7f2ab000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.362+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0003b70 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.362+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab0003d10 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.362+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab80c94b0 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.362+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00052d0 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.362+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab815f670 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.363+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab80c99d0 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.363+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ab0005470 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.364+0000 7f2ab4ff9640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2aa406c8f0 0x7f2aa406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.364+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f2ab008b500 con 0x7f2ab815eae0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.364+0000 7f2ab7fff640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2aa406c8f0 0x7f2aa406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.364+0000 7f2ab7fff640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2aa406c8f0 0x7f2aa406edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8001000 tx=0x7f2aa8001da0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.368+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ab80c99d0 con 0x7f2ab815eae0 2024-04-15T15:15:35.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.553+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ab80689e0 con 0x7f2ab815eae0 2024-04-15T15:15:35.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.554+0000 7f2ab4ff9640 1 -- 172.21.15.46:0/3087152704 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12836 (secure 0 0 0) 0x7f2ab00596c0 con 0x7f2ab815eae0 2024-04-15T15:15:35.556 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:35.556 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:28.519563+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:35.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.556+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2aa406c8f0 msgr2=0x7f2aa406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.556+0000 7f2abe61b640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2aa406c8f0 0x7f2aa406edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8001000 tx=0x7f2aa8001da0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.556+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 msgr2=0x7f2ab80ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.556+0000 7f2abe61b640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 0x7f2ab80ca2f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000c6e0 tx=0x7f2ab000cba0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 shutdown_connections 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2aa406c8f0 0x7f2aa406edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ab815eae0 0x7f2ab80ca2f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ab815e700 0x7f2ab80cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 --2- 172.21.15.46:0/3087152704 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ab80d2360 0x7f2ab8153870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 >> 172.21.15.46:0/3087152704 conn(0x7f2ab808bf60 msgr2=0x7f2ab80d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 shutdown_connections 2024-04-15T15:15:35.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.557+0000 7f2abe61b640 1 -- 172.21.15.46:0/3087152704 wait complete. 2024-04-15T15:15:35.569 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:15:35.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:35.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.675+0000 7f94c8c7b640 1 Processor -- start 2024-04-15T15:15:35.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.675+0000 7f94c8c7b640 1 -- start start 2024-04-15T15:15:35.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c8c7b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f94c40d5150 0x7f94c40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c8c7b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94c415eb70 con 0x7f94c40d15e0 2024-04-15T15:15:35.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c8c7b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94c40d3c80 con 0x7f94c415e790 2024-04-15T15:15:35.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c8c7b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f94c40d3790 con 0x7f94c40d5150 2024-04-15T15:15:35.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c1d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f94c40d15e0 0x7f94c40d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55480/0 (socket says 172.21.15.46:55480) 2024-04-15T15:15:35.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c1d9b640 1 -- 172.21.15.46:0/2000119523 learned_addr learned my addr 172.21.15.46:0/2000119523 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c2d9d640 1 --2- 172.21.15.46:0/2000119523 >> v2:172.21.15.78:3300/0 conn(0x7f94c40d5150 0x7f94c40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143056031 0 0) 0x7f94c415eb70 con 0x7f94c40d15e0 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94b0003520 con 0x7f94c40d15e0 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233046431 0 0) 0x7f94c40d3c80 con 0x7f94c415e790 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94c415eb70 con 0x7f94c415e790 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 498279856 0 0) 0x7f94b0003520 con 0x7f94c40d15e0 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 >> v2:172.21.15.78:3300/0 conn(0x7f94c40d5150 msgr2=0x7f94c40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 --2- 172.21.15.46:0/2000119523 >> v2:172.21.15.78:3300/0 conn(0x7f94c40d5150 0x7f94c40d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 >> v1:172.21.15.78:6789/0 conn(0x7f94c415e790 legacy=0x7f94c40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c40d3f60 con 0x7f94c40d15e0 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f94ac001670 con 0x7f94c40d15e0 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 >> v1:172.21.15.46:6789/0 conn(0x7f94c40d15e0 legacy=0x7f94c40d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0003b40 0x7f94b0003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94b0005230 0x7f94b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94c40d3790 con 0x7f94b0005230 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f94b000c190 con 0x7f94b0003b40 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f94b000c390 con 0x7f94b0004800 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.676+0000 7f94c2d9d640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94b0005230 0x7f94b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c259c640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0003b40 0x7f94b0003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c1d9b640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 -- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94b0004800 msgr2=0x7f94b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94b0004800 0x7f94b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 -- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0003b40 msgr2=0x7f94b0003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0003b40 0x7f94b0003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 -- 172.21.15.46:0/2000119523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b0003520 con 0x7f94b0005230 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 -- 172.21.15.46:0/2000119523 reap_dead start 2024-04-15T15:15:35.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.677+0000 7f94c2d9d640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94b0005230 0x7f94b000b9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009fc0 tx=0x7f94b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a645ae0cb8376cdb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4009c10 con 0x7f94b0005230 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b402cca0 con 0x7f94b0005230 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c159a640 1 -- 172.21.15.46:0/2000119523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4003bd0 con 0x7f94b0005230 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 -- 172.21.15.46:0/2000119523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94b0005230 msgr2=0x7f94b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94b0005230 0x7f94b000b9c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009fc0 tx=0x7f94b0005830 comp rx=0 tx=0).stop 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 -- 172.21.15.46:0/2000119523 shutdown_connections 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/2000119523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94b0005230 0x7f94b000b9c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 -- 172.21.15.46:0/2000119523 >> 172.21.15.46:0/2000119523 conn(0x7f94c4067b40 msgr2=0x7f94c4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 -- 172.21.15.46:0/2000119523 shutdown_connections 2024-04-15T15:15:35.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.678+0000 7f94c8c7b640 1 -- 172.21.15.46:0/2000119523 wait complete. 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 Processor -- start 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 -- start start 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0005230 0x7f94c41f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 0x7f94c40abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94c40cd6f0 0x7f94c40cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f94c40d1a90 con 0x7f94c4152d40 2024-04-15T15:15:35.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f94c40d45b0 con 0x7f94b0005230 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c8c7b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f94c40d1c10 con 0x7f94c40cd6f0 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 0x7f94c40abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 0x7f94c40abd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49748/0 (socket says 172.21.15.46:49748) 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.680+0000 7f94c1d9b640 1 -- 172.21.15.46:0/678076385 learned_addr learned my addr 172.21.15.46:0/678076385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c259c640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0005230 0x7f94c41f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c2d9d640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94c40cd6f0 0x7f94c40cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c1d9b640 1 -- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94c40cd6f0 msgr2=0x7f94c40cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c1d9b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94c40cd6f0 0x7f94c40cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c1d9b640 1 -- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0005230 msgr2=0x7f94c41f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c1d9b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0005230 0x7f94c41f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c1d9b640 1 -- 172.21.15.46:0/678076385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c40c9330 con 0x7f94c4152d40 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c1d9b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 0x7f94c40abd10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00e780 tx=0x7f94ac00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00c660 con 0x7f94c4152d40 2024-04-15T15:15:35.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ac018070 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c40c9610 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00ca20 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.681+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c40c9bd0 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.682+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c40689e0 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.682+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f94ac00cbc0 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.683+0000 7f94baffd640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f94a406c8f0 0x7f94a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.683+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f94ac08b520 con 0x7f94c4152d40 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.683+0000 7f94c259c640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f94a406c8f0 0x7f94a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.683+0000 7f94c259c640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f94a406c8f0 0x7f94a406edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0010e0 tx=0x7f94bc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.687+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94ac059d30 con 0x7f94c4152d40 2024-04-15T15:15:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.873+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94c40d04b0 con 0x7f94c4152d40 2024-04-15T15:15:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.873+0000 7f94baffd640 1 -- 172.21.15.46:0/678076385 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f94ac059630 con 0x7f94c4152d40 2024-04-15T15:15:35.875 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:15:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f94a406c8f0 msgr2=0x7f94a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f94a406c8f0 0x7f94a406edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f94bc0010e0 tx=0x7f94bc009040 comp rx=0 tx=0).stop 2024-04-15T15:15:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 msgr2=0x7f94c40abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 0x7f94c40abd10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00e780 tx=0x7f94ac00ec40 comp rx=0 tx=0).stop 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 shutdown_connections 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f94a406c8f0 0x7f94a406edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f94c40cd6f0 0x7f94c40cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94c4152d40 0x7f94c40abd10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 --2- 172.21.15.46:0/678076385 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f94b0005230 0x7f94c41f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 >> 172.21.15.46:0/678076385 conn(0x7f94c4067b40 msgr2=0x7f94c40b90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 shutdown_connections 2024-04-15T15:15:35.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.875+0000 7f94c8c7b640 1 -- 172.21.15.46:0/678076385 wait complete. 2024-04-15T15:15:35.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:15:35.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.991+0000 7f6201ed9640 1 Processor -- start 2024-04-15T15:15:35.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.991+0000 7f6201ed9640 1 -- start start 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.991+0000 7f6201ed9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0af340 0x7f61fc0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.991+0000 7f6201ed9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc0af920 con 0x7f61fc15ecb0 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.991+0000 7f6201ed9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61fc0afc00 con 0x7f61fc0b1620 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.991+0000 7f6201ed9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f61fc15f090 con 0x7f61fc0af340 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fb7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f61fc15ecb0 0x7f61fc0b0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55496/0 (socket says 172.21.15.46:55496) 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fb7fe640 1 -- 172.21.15.46:0/3290234777 learned_addr learned my addr 172.21.15.46:0/3290234777 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:35.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61faffd640 1 --2- 172.21.15.46:0/3290234777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0af340 0x7f61fc0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 264835110 0 0) 0x7f61fc0af920 con 0x7f61fc15ecb0 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61e4003520 con 0x7f61fc15ecb0 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3026843501 0 0) 0x7f61fc0afc00 con 0x7f61fc0b1620 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61fc0af920 con 0x7f61fc0b1620 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61faffd640 1 -- 172.21.15.46:0/3290234777 >> v1:172.21.15.78:6789/0 conn(0x7f61fc0b1620 legacy=0x7f61fc0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61faffd640 1 -- 172.21.15.46:0/3290234777 >> v1:172.21.15.46:6789/0 conn(0x7f61fc15ecb0 legacy=0x7f61fc0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2187006015 0 0) 0x7f61e4003520 con 0x7f61fc15ecb0 2024-04-15T15:15:35.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.992+0000 7f61faffd640 1 -- 172.21.15.46:0/3290234777 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc0afee0 con 0x7f61fc0af340 2024-04-15T15:15:35.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.993+0000 7f61faffd640 1 --2- 172.21.15.46:0/3290234777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0af340 0x7f61fc0b4350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097d0 tx=0x7f61f002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50ef58915f83682d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.993+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec001670 con 0x7f61fc0af340 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.993+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f0037070 con 0x7f61fc0af340 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.993+0000 7f61fa7fc640 1 -- 172.21.15.46:0/3290234777 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f00317a0 con 0x7f61fc0af340 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 -- 172.21.15.46:0/3290234777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0af340 msgr2=0x7f61fc0b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 --2- 172.21.15.46:0/3290234777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0af340 0x7f61fc0b4350 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f61f00097d0 tx=0x7f61f002b8a0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 -- 172.21.15.46:0/3290234777 shutdown_connections 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 --2- 172.21.15.46:0/3290234777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0af340 0x7f61fc0b4350 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 -- 172.21.15.46:0/3290234777 >> 172.21.15.46:0/3290234777 conn(0x7f61fc0b6870 msgr2=0x7f61fc0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 -- 172.21.15.46:0/3290234777 shutdown_connections 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.994+0000 7f6201ed9640 1 -- 172.21.15.46:0/3290234777 wait complete. 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.995+0000 7f6201ed9640 1 Processor -- start 2024-04-15T15:15:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.995+0000 7f6201ed9640 1 -- start start 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 0x7f61fc1580f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0b1620 0x7f61fc09e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61fc15ecb0 0x7f61fc09b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc0ceb20 con 0x7f61fc0af340 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc0cdf80 con 0x7f61fc15ecb0 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f61fc0ceca0 con 0x7f61fc0b1620 2024-04-15T15:15:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 0x7f61fc1580f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 0x7f61fc1580f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49764/0 (socket says 172.21.15.46:49764) 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 -- 172.21.15.46:0/2660209049 learned_addr learned my addr 172.21.15.46:0/2660209049 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61faffd640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0b1620 0x7f61fc09e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fbfff640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61fc15ecb0 0x7f61fc09b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 -- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0b1620 msgr2=0x7f61fc09e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0b1620 0x7f61fc09e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 -- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61fc15ecb0 msgr2=0x7f61fc09b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61fc15ecb0 0x7f61fc09b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 -- 172.21.15.46:0/2660209049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc09eb90 con 0x7f61fc0af340 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fbfff640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61fc15ecb0 0x7f61fc09b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61fb7fe640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 0x7f61fc1580f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c550 tx=0x7f61ec00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec017070 con 0x7f61fc0af340 2024-04-15T15:15:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc09ee70 con 0x7f61fc0af340 2024-04-15T15:15:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc158630 con 0x7f61fc0af340 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ec005ce0 con 0x7f61fc0af340 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.996+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec007270 con 0x7f61fc0af340 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.998+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61dc005290 con 0x7f61fc0af340 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.998+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f61ec01a070 con 0x7f61fc0af340 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.998+0000 7f61e3fff640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61e806c8f0 0x7f61e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.998+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f61ec08ade0 con 0x7f61fc0af340 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.998+0000 7f61faffd640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61e806c8f0 0x7f61e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:35.999+0000 7f61faffd640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61e806c8f0 0x7f61e806edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61f0036000 tx=0x7f61f0008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.003+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61ec058f20 con 0x7f61fc0af340 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.189+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f61dc005850 con 0x7f61fc0af340 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.189+0000 7f61e3fff640 1 -- 172.21.15.46:0/2660209049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f61ec05d220 con 0x7f61fc0af340 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:15:36.191 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-04-15T15:15:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61e806c8f0 msgr2=0x7f61e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61e806c8f0 0x7f61e806edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61f0036000 tx=0x7f61f0008030 comp rx=0 tx=0).stop 2024-04-15T15:15:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 msgr2=0x7f61fc1580f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 0x7f61fc1580f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c550 tx=0x7f61ec00ca10 comp rx=0 tx=0).stop 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 shutdown_connections 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61e806c8f0 0x7f61e806edb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61fc15ecb0 0x7f61fc09b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61fc0b1620 0x7f61fc09e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 --2- 172.21.15.46:0/2660209049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61fc0af340 0x7f61fc1580f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 >> 172.21.15.46:0/2660209049 conn(0x7f61fc0b6870 msgr2=0x7f61fc0704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 shutdown_connections 2024-04-15T15:15:36.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.191+0000 7f6201ed9640 1 -- 172.21.15.46:0/2660209049 wait complete. 2024-04-15T15:15:36.204 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:15:36.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:15:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.311+0000 7f0a222c8640 1 Processor -- start 2024-04-15T15:15:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.311+0000 7f0a222c8640 1 -- start start 2024-04-15T15:15:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a222c8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a1c15e930 0x7f0a1c0ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a222c8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0a1c0b77b0 0x7f0a1c0d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a222c8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a1c15ed10 con 0x7f0a1c0d16e0 2024-04-15T15:15:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a222c8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0a1c0b8bc0 con 0x7f0a1c15e930 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a222c8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0a1c0b7fb0 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a1c15e930 0x7f0a1c0ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0a1c15e930 0x7f0a1c0ba260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49768/0 (socket says 172.21.15.46:49768) 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 -- 172.21.15.46:0/1725372890 learned_addr learned my addr 172.21.15.46:0/1725372890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a13fff640 1 --2- 172.21.15.46:0/1725372890 >> v2:172.21.15.78:3300/0 conn(0x7f0a1c0b77b0 0x7f0a1c0d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 -- 172.21.15.46:0/1725372890 >> v2:172.21.15.78:3300/0 conn(0x7f0a1c0b77b0 msgr2=0x7f0a1c0d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 --2- 172.21.15.46:0/1725372890 >> v2:172.21.15.78:3300/0 conn(0x7f0a1c0b77b0 0x7f0a1c0d2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 -- 172.21.15.46:0/1725372890 >> v1:172.21.15.78:6789/0 conn(0x7f0a1c0d16e0 legacy=0x7f0a1c0b9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.312+0000 7f0a1bfff640 1 -- 172.21.15.46:0/1725372890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c0b8290 con 0x7f0a1c15e930 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.313+0000 7f0a1bfff640 1 --2- 172.21.15.46:0/1725372890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ba260 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c06dc50 tx=0x7f0a0c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debef4501a3b45be server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.313+0000 7f0a1b7fe640 1 -- 172.21.15.46:0/1725372890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c034070 con 0x7f0a1c15e930 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.313+0000 7f0a1b7fe640 1 -- 172.21.15.46:0/1725372890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0c030860 con 0x7f0a1c15e930 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.313+0000 7f0a1b7fe640 1 -- 172.21.15.46:0/1725372890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c030c30 con 0x7f0a1c15e930 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 -- 172.21.15.46:0/1725372890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c15e930 msgr2=0x7f0a1c0ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 --2- 172.21.15.46:0/1725372890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ba260 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c06dc50 tx=0x7f0a0c02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 -- 172.21.15.46:0/1725372890 shutdown_connections 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 --2- 172.21.15.46:0/1725372890 >> v2:172.21.15.78:3300/0 conn(0x7f0a1c0b77b0 0x7f0a1c0d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 --2- 172.21.15.46:0/1725372890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ba260 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 -- 172.21.15.46:0/1725372890 >> 172.21.15.46:0/1725372890 conn(0x7f0a1c067b10 msgr2=0x7f0a1c067f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 -- 172.21.15.46:0/1725372890 shutdown_connections 2024-04-15T15:15:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.314+0000 7f0a222c8640 1 -- 172.21.15.46:0/1725372890 wait complete. 2024-04-15T15:15:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.315+0000 7f0a222c8640 1 Processor -- start 2024-04-15T15:15:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.315+0000 7f0a222c8640 1 -- start start 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a222c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 0x7f0a1c150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a222c8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0a1c0d16e0 0x7f0a1c0cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a222c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a222c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c0d2fa0 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a222c8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c0b8850 con 0x7f0a1c15e930 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a222c8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0a1c0b7bf0 con 0x7f0a1c0d16e0 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 0x7f0a1c150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 0x7f0a1c150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49774/0 (socket says 172.21.15.46:49774) 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 -- 172.21.15.46:0/356027818 learned_addr learned my addr 172.21.15.46:0/356027818 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:36.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a13fff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0a1c0d16e0 0x7f0a1c0cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a20865640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 -- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0a1c0d16e0 msgr2=0x7f0a1c0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0a1c0d16e0 0x7f0a1c0cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 -- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a1c15e930 msgr2=0x7f0a1c0ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a1bfff640 1 -- 172.21.15.46:0/356027818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c15f060 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.316+0000 7f0a13fff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0a1c0d16e0 0x7f0a1c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.317+0000 7f0a1bfff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 0x7f0a1c150ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c02b9e0 tx=0x7f0a0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.317+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c034040 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.317+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0c039d90 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.317+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1c15f340 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.317+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c043610 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.317+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1c0cbd50 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.318+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a1c0689e0 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.318+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a0c037070 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.319+0000 7f0a197fa640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f09fc06c8f0 0x7f09fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.319+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0a0c0ad8b0 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.319+0000 7f0a13fff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f09fc06c8f0 0x7f09fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.320+0000 7f0a13fff640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f09fc06c8f0 0x7f09fc06edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c1f7930 tx=0x7f0a04008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.323+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a0c03e070 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.508+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f0a1c0b9a10 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.537+0000 7f0a197fa640 1 -- 172.21.15.46:0/356027818 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f0a0c07c0c0 con 0x7f0a1c0b77b0 2024-04-15T15:15:36.539 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-04-15T15:15:36.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f09fc06c8f0 msgr2=0x7f09fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f09fc06c8f0 0x7f09fc06edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c1f7930 tx=0x7f0a04008040 comp rx=0 tx=0).stop 2024-04-15T15:15:36.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 msgr2=0x7f0a1c150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 0x7f0a1c150ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c02b9e0 tx=0x7f0a0c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 shutdown_connections 2024-04-15T15:15:36.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f09fc06c8f0 0x7f09fc06edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0a1c15e930 0x7f0a1c0ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0a1c0d16e0 0x7f0a1c0cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 --2- 172.21.15.46:0/356027818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a1c0b77b0 0x7f0a1c150ca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 >> 172.21.15.46:0/356027818 conn(0x7f0a1c067b10 msgr2=0x7f0a1c0d3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 shutdown_connections 2024-04-15T15:15:36.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.539+0000 7f0a222c8640 1 -- 172.21.15.46:0/356027818 wait complete. 2024-04-15T15:15:36.555 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-15T15:15:36.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.665+0000 7f74c99db640 1 Processor -- start 2024-04-15T15:15:36.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.665+0000 7f74c99db640 1 -- start start 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c99db640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c40d2360 0x7f74c40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c99db640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74c415eae0 0x7f74c40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c99db640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74c40b47a0 con 0x7f74c415e700 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c99db640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f74c40b53b0 con 0x7f74c415eae0 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c99db640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40b4a80 con 0x7f74c40d2360 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74c415eae0 0x7f74c40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f74c415e700 0x7f74c40b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55502/0 (socket says 172.21.15.46:55502) 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f74c415eae0 0x7f74c40d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49782/0 (socket says 172.21.15.46:49782) 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c2ffd640 1 -- 172.21.15.46:0/1297810004 learned_addr learned my addr 172.21.15.46:0/1297810004 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c27fc640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c40d2360 0x7f74c40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c37fe640 1 -- 172.21.15.46:0/1297810004 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c40d2360 msgr2=0x7f74c40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c37fe640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c40d2360 0x7f74c40d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c37fe640 1 -- 172.21.15.46:0/1297810004 >> v1:172.21.15.46:6789/0 conn(0x7f74c415e700 legacy=0x7f74c40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c37fe640 1 -- 172.21.15.46:0/1297810004 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c40b4d60 con 0x7f74c415eae0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c1ffb640 1 -- 172.21.15.46:0/1297810004 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 867222596 0 0) 0x7f74c40b47a0 con 0x7f74c415e700 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.666+0000 7f74c27fc640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c40d2360 0x7f74c40d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.667+0000 7f74c37fe640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415eae0 0x7f74c40d30a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74b40097a0 tx=0x7f74b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda9a25be1217674 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.667+0000 7f74c1ffb640 1 -- 172.21.15.46:0/1297810004 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4035020 con 0x7f74c415eae0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.667+0000 7f74c1ffb640 1 -- 172.21.15.46:0/1297810004 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b4031830 con 0x7f74c415eae0 2024-04-15T15:15:36.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.667+0000 7f74c1ffb640 1 -- 172.21.15.46:0/1297810004 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4031c00 con 0x7f74c415eae0 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 -- 172.21.15.46:0/1297810004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415eae0 msgr2=0x7f74c40d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415eae0 0x7f74c40d30a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f74b40097a0 tx=0x7f74b402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 -- 172.21.15.46:0/1297810004 shutdown_connections 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415eae0 0x7f74c40d30a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 --2- 172.21.15.46:0/1297810004 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c40d2360 0x7f74c40d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 -- 172.21.15.46:0/1297810004 >> 172.21.15.46:0/1297810004 conn(0x7f74c408bf60 msgr2=0x7f74c40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 -- 172.21.15.46:0/1297810004 shutdown_connections 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.668+0000 7f74c99db640 1 -- 172.21.15.46:0/1297810004 wait complete. 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.669+0000 7f74c99db640 1 Processor -- start 2024-04-15T15:15:36.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.669+0000 7f74c99db640 1 -- start start 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c99db640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c40d2360 0x7f74c4157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c99db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 0x7f74c406e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c99db640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c415eae0 0x7f74c40c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c99db640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74c40d1d40 con 0x7f74c415e700 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c99db640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f74c40b4450 con 0x7f74c40d2360 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c99db640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40d1ec0 con 0x7f74c415eae0 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 0x7f74c406e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 0x7f74c406e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49784/0 (socket says 172.21.15.46:49784) 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 -- 172.21.15.46:0/1522880979 learned_addr learned my addr 172.21.15.46:0/1522880979 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c37fe640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c415eae0 0x7f74c40c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c2ffd640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c40d2360 0x7f74c4157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 -- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c415eae0 msgr2=0x7f74c40c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c415eae0 0x7f74c40c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 -- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c40d2360 msgr2=0x7f74c4157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c40d2360 0x7f74c4157a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 -- 172.21.15.46:0/1522880979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c40c8490 con 0x7f74c415e700 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c37fe640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c415eae0 0x7f74c40c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.670+0000 7f74c27fc640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 0x7f74c406e000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e7e0 tx=0x7f74ac00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.671+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac00f040 con 0x7f74c415e700 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.671+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74ac004030 con 0x7f74c415e700 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.671+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c40c8730 con 0x7f74c415e700 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.671+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac004400 con 0x7f74c415e700 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.671+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c40cc110 con 0x7f74c415e700 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.672+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c40689e0 con 0x7f74c415e700 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.672+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74ac00c6d0 con 0x7f74c415e700 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.673+0000 7f74bbfff640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f74a406c8f0 0x7f74a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.673+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f74ac08b700 con 0x7f74c415e700 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.673+0000 7f74c2ffd640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f74a406c8f0 0x7f74a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.674+0000 7f74c2ffd640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f74a406c8f0 0x7f74a406edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74c40b6a00 tx=0x7f74bc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.677+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74ac017020 con 0x7f74c415e700 2024-04-15T15:15:36.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:36.864+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f74c40ba4c0 con 0x7f74c415e700 2024-04-15T15:15:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.542+0000 7f74bbfff640 1 -- 172.21.15.46:0/1522880979 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f74ac059d10 con 0x7f74c415e700 2024-04-15T15:15:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f74a406c8f0 msgr2=0x7f74a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f74a406c8f0 0x7f74a406edb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74c40b6a00 tx=0x7f74bc009040 comp rx=0 tx=0).stop 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 msgr2=0x7f74c406e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 0x7f74c406e000 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e7e0 tx=0x7f74ac00eca0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 shutdown_connections 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f74a406c8f0 0x7f74a406edb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74c415eae0 0x7f74c40c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74c415e700 0x7f74c406e000 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74ac00e7e0 tx=0x7f74ac00eca0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 --2- 172.21.15.46:0/1522880979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74c40d2360 0x7f74c4157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.546+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 >> 172.21.15.46:0/1522880979 conn(0x7f74c408bf60 msgr2=0x7f74c40cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.547+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 shutdown_connections 2024-04-15T15:15:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.548+0000 7f74c99db640 1 -- 172.21.15.46:0/1522880979 wait complete. 2024-04-15T15:15:37.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:15:37.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.689+0000 7fe37502a640 1 Processor -- start 2024-04-15T15:15:37.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.689+0000 7fe37502a640 1 -- start start 2024-04-15T15:15:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe37502a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3700d3270 0x7fe3700cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe37502a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3700b47a0 con 0x7fe3700d2360 2024-04-15T15:15:37.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe37502a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3700b4a80 con 0x7fe37015e700 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe37502a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe3700b53b0 con 0x7fe3700d3270 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe367fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe3700d2360 0x7fe3700d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55512/0 (socket says 172.21.15.46:55512) 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe367fff640 1 -- 172.21.15.46:0/1038139356 learned_addr learned my addr 172.21.15.46:0/1038139356 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36f59e640 1 --2- 172.21.15.46:0/1038139356 >> v2:172.21.15.78:3300/0 conn(0x7fe3700d3270 0x7fe3700cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480665236 0 0) 0x7fe3700b47a0 con 0x7fe3700d2360 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe35c003520 con 0x7fe3700d2360 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2374302862 0 0) 0x7fe35c003520 con 0x7fe3700d2360 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 >> v2:172.21.15.78:3300/0 conn(0x7fe3700d3270 msgr2=0x7fe3700cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 --2- 172.21.15.46:0/1038139356 >> v2:172.21.15.78:3300/0 conn(0x7fe3700d3270 0x7fe3700cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 >> v1:172.21.15.78:6789/0 conn(0x7fe37015e700 legacy=0x7fe3700b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3700b4d60 con 0x7fe3700d2360 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fe358001670 con 0x7fe3700d2360 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 >> v1:172.21.15.46:6789/0 conn(0x7fe3700d2360 legacy=0x7fe3700d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe35c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35c004800 0x7fe35c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe35c0051c0 0x7fe35c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700b53b0 con 0x7fe35c0039b0 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c00c090 con 0x7fe35c0051c0 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe35c00c290 con 0x7fe35c004800 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36f59e640 1 --2- 172.21.15.46:0/1038139356 >> v2:172.21.15.78:3300/0 conn(0x7fe3700d3270 0x7fe3700cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.690+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe35c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe367fff640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35c004800 0x7fe35c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36f59e640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe35c0051c0 0x7fe35c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36ed9d640 1 -- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35c004800 msgr2=0x7fe35c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35c004800 0x7fe35c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36ed9d640 1 -- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe35c0051c0 msgr2=0x7fe35c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe35c0051c0 0x7fe35c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36ed9d640 1 -- 172.21.15.46:0/1038139356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35c003520 con 0x7fe35c0039b0 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36f59e640 1 -- 172.21.15.46:0/1038139356 reap_dead start 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.691+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe35c003df0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe360001770 tx=0x7fe35c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6201962f248bdce server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3600013c0 con 0x7fe35c0039b0 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3600052d0 con 0x7fe35c0039b0 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe36e59c640 1 -- 172.21.15.46:0/1038139356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360003640 con 0x7fe35c0039b0 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 -- 172.21.15.46:0/1038139356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 msgr2=0x7fe35c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe35c003df0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe360001770 tx=0x7fe35c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 -- 172.21.15.46:0/1038139356 shutdown_connections 2024-04-15T15:15:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 --2- 172.21.15.46:0/1038139356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe35c003df0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 -- 172.21.15.46:0/1038139356 >> 172.21.15.46:0/1038139356 conn(0x7fe37008bf60 msgr2=0x7fe3700d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 -- 172.21.15.46:0/1038139356 shutdown_connections 2024-04-15T15:15:37.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.692+0000 7fe37502a640 1 -- 172.21.15.46:0/1038139356 wait complete. 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 Processor -- start 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 -- start start 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe3700b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3700b3110 0x7fe3700b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3700b3ad0 0x7fe37014d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700d1d40 con 0x7fe35c0039b0 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700b4450 con 0x7fe3700b3ad0 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe37502a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe3700d1ec0 con 0x7fe3700b3110 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe36ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe3700b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe36ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe3700b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49808/0 (socket says 172.21.15.46:49808) 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.694+0000 7fe36ed9d640 1 -- 172.21.15.46:0/3759932734 learned_addr learned my addr 172.21.15.46:0/3759932734 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:37.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe367fff640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3700b3110 0x7fe3700b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36f59e640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3700b3ad0 0x7fe37014d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36ed9d640 1 -- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3700b3110 msgr2=0x7fe3700b3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3700b3110 0x7fe3700b3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36ed9d640 1 -- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3700b3ad0 msgr2=0x7fe37014d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3700b3ad0 0x7fe37014d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36ed9d640 1 -- 172.21.15.46:0/3759932734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3700b3f50 con 0x7fe35c0039b0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe367fff640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3700b3110 0x7fe3700b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36f59e640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3700b3ad0 0x7fe37014d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe36ed9d640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe3700b72d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe360001890 tx=0x7fe360001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360041070 con 0x7fe35c0039b0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3600050d0 con 0x7fe35c0039b0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3700cc510 con 0x7fe35c0039b0 2024-04-15T15:15:37.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360007470 con 0x7fe35c0039b0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.695+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3700ccad0 con 0x7fe35c0039b0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.696+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3700689e0 con 0x7fe35c0039b0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.697+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe360044070 con 0x7fe35c0039b0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.697+0000 7fe3677fe640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35006c920 0x7fe35006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.697+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fe3600ad5a0 con 0x7fe35c0039b0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.697+0000 7fe367fff640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35006c920 0x7fe35006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.698+0000 7fe367fff640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35006c920 0x7fe35006ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe370161fd0 tx=0x7fe358009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.702+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe36007ba30 con 0x7fe35c0039b0 2024-04-15T15:15:37.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:37.924+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe37015b180 con 0x7fe35c0039b0 2024-04-15T15:15:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.553+0000 7fe3677fe640 1 -- 172.21.15.46:0/3759932734 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fe36007b330 con 0x7fe35c0039b0 2024-04-15T15:15:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2024-04-15T15:15:38.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.562+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35006c920 msgr2=0x7fe35006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.562+0000 7fe37502a640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35006c920 0x7fe35006ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe370161fd0 tx=0x7fe358009040 comp rx=0 tx=0).stop 2024-04-15T15:15:38.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.562+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 msgr2=0x7fe3700b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.562+0000 7fe37502a640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe3700b72d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe360001890 tx=0x7fe360001170 comp rx=0 tx=0).stop 2024-04-15T15:15:38.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 shutdown_connections 2024-04-15T15:15:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35006c920 0x7fe35006ede0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3700b3ad0 0x7fe37014d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3700b3110 0x7fe3700b3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 --2- 172.21.15.46:0/3759932734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35c0039b0 0x7fe3700b72d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 >> 172.21.15.46:0/3759932734 conn(0x7fe37008bf60 msgr2=0x7fe3700ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.564+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 shutdown_connections 2024-04-15T15:15:38.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.565+0000 7fe37502a640 1 -- 172.21.15.46:0/3759932734 wait complete. 2024-04-15T15:15:38.586 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-15T15:15:38.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-15T15:15:38.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 Processor -- start 2024-04-15T15:15:38.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 -- start start 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f029415ea40 0x7f02940b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f029415ee20 0x7f02940b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02940cff70 con 0x7f02940b65c0 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f02940b6a10 con 0x7f029415ee20 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f029ab7c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f02940d0250 con 0x7f029415ea40 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.699+0000 7f0293fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f029415ee20 0x7f02940b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0293fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f029415ee20 0x7f02940b6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49814/0 (socket says 172.21.15.46:49814) 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0299119640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f02940b65c0 0x7f02940d2470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55526/0 (socket says 172.21.15.46:55526) 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0293fff640 1 -- 172.21.15.46:0/4246976399 learned_addr learned my addr 172.21.15.46:0/4246976399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:38.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0298918640 1 --2- 172.21.15.46:0/4246976399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f029415ea40 0x7f02940b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0293fff640 1 -- 172.21.15.46:0/4246976399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f029415ea40 msgr2=0x7f02940b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0293fff640 1 --2- 172.21.15.46:0/4246976399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f029415ea40 0x7f02940b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0293fff640 1 -- 172.21.15.46:0/4246976399 >> v1:172.21.15.46:6789/0 conn(0x7f02940b65c0 legacy=0x7f02940d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.700+0000 7f0293fff640 1 -- 172.21.15.46:0/4246976399 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02940d0530 con 0x7f029415ee20 2024-04-15T15:15:38.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f0293fff640 1 --2- 172.21.15.46:0/4246976399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02940b6080 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0288009870 tx=0x7f028802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218e692b92fc4ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f02937fe640 1 -- 172.21.15.46:0/4246976399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028802f840 con 0x7f029415ee20 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f02937fe640 1 -- 172.21.15.46:0/4246976399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f028802f9e0 con 0x7f029415ee20 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f02937fe640 1 -- 172.21.15.46:0/4246976399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f028802fdb0 con 0x7f029415ee20 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 -- 172.21.15.46:0/4246976399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 msgr2=0x7f02940b6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 --2- 172.21.15.46:0/4246976399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02940b6080 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0288009870 tx=0x7f028802ba20 comp rx=0 tx=0).stop 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 -- 172.21.15.46:0/4246976399 shutdown_connections 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 --2- 172.21.15.46:0/4246976399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02940b6080 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 --2- 172.21.15.46:0/4246976399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f029415ea40 0x7f02940b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 -- 172.21.15.46:0/4246976399 >> 172.21.15.46:0/4246976399 conn(0x7f029408bf60 msgr2=0x7f02940d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 -- 172.21.15.46:0/4246976399 shutdown_connections 2024-04-15T15:15:38.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.701+0000 7f029ab7c640 1 -- 172.21.15.46:0/4246976399 wait complete. 2024-04-15T15:15:38.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.702+0000 7f029ab7c640 1 Processor -- start 2024-04-15T15:15:38.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.702+0000 7f029ab7c640 1 -- start start 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f029ab7c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02940b65c0 0x7f02941f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f029ab7c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f029415ea40 0x7f0294160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f029ab7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02941f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f029ab7c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02940b38c0 con 0x7f029415ee20 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f029ab7c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f02940d0af0 con 0x7f029415ea40 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f029ab7c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f02940b3a40 con 0x7f02940b65c0 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f0299119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02941f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f0299119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02941f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49826/0 (socket says 172.21.15.46:49826) 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.703+0000 7f0299119640 1 -- 172.21.15.46:0/20268916 learned_addr learned my addr 172.21.15.46:0/20268916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0298918640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02940b65c0 0x7f02941f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0293fff640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f029415ea40 0x7f0294160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0299119640 1 -- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02940b65c0 msgr2=0x7f02941f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0299119640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02940b65c0 0x7f02941f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0299119640 1 -- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f029415ea40 msgr2=0x7f0294160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0299119640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f029415ea40 0x7f0294160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0299119640 1 -- 172.21.15.46:0/20268916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02941f7690 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f0299119640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02941f7150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f028400c400 tx=0x7f028400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840074e0 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0284007680 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0294160ab0 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02840049e0 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.704+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0294161070 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.705+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02940689e0 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.706+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f028401a020 con 0x7f029415ee20 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.706+0000 7f02917fa640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028006c8f0 0x7f028006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.706+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f028408b330 con 0x7f029415ee20 2024-04-15T15:15:38.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.706+0000 7f0298918640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028006c8f0 0x7f028006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.707+0000 7f0298918640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028006c8f0 0x7f028006edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f028c009a60 tx=0x7f028c003990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.710+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f02941f7690 con 0x7f029415ee20 2024-04-15T15:15:38.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:38.908+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f0294075000 con 0x7f029415ee20 2024-04-15T15:15:38.971 INFO:tasks.ceph.mds.b.smithi078.stderr:2024-04-15T15:15:38.969+0000 7fabe1623640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:15:39.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.553+0000 7f02917fa640 1 -- 172.21.15.46:0/20268916 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f0284059110 con 0x7f029415ee20 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028006c8f0 msgr2=0x7f028006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028006c8f0 0x7f028006edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f028c009a60 tx=0x7f028c003990 comp rx=0 tx=0).stop 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 msgr2=0x7f02941f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02941f7150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f028400c400 tx=0x7f028400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 shutdown_connections 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028006c8f0 0x7f028006edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029415ee20 0x7f02941f7150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f029415ea40 0x7f0294160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 --2- 172.21.15.46:0/20268916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02940b65c0 0x7f02941f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 >> 172.21.15.46:0/20268916 conn(0x7f029408bf60 msgr2=0x7f02940b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 shutdown_connections 2024-04-15T15:15:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.557+0000 7f029ab7c640 1 -- 172.21.15.46:0/20268916 wait complete. 2024-04-15T15:15:39.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:15:39.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:39.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.690+0000 7f850eb9d640 1 Processor -- start 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.690+0000 7f850eb9d640 1 -- start start 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850eb9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85080d1310 0x7f85080d1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850eb9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85080b74f0 0x7f85080b7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850eb9d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f850815eb40 con 0x7f850815e760 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850eb9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f85080d1cb0 con 0x7f85080b74f0 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850eb9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f85080d2f90 con 0x7f85080d1310 2024-04-15T15:15:39.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850e39c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85080b74f0 0x7f85080b7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850e39c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85080b74f0 0x7f85080b7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43764/0 (socket says 172.21.15.46:43764) 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850e39c640 1 -- 172.21.15.46:0/1938318031 learned_addr learned my addr 172.21.15.46:0/1938318031 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.691+0000 7f850d39a640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85080d1310 0x7f85080d1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850e39c640 1 -- 172.21.15.46:0/1938318031 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85080d1310 msgr2=0x7f85080d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850e39c640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85080d1310 0x7f85080d1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850e39c640 1 -- 172.21.15.46:0/1938318031 >> v1:172.21.15.78:6789/0 conn(0x7f850815e760 legacy=0x7f85080d24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850e39c640 1 -- 172.21.15.46:0/1938318031 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85080d3270 con 0x7f85080b74f0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850d39a640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85080d1310 0x7f85080d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850e39c640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080b74f0 0x7f85080b7980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009e50 tx=0x7f84f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceb0c14a3209c63 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850cb99640 1 -- 172.21.15.46:0/1938318031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f802ca80 con 0x7f85080b74f0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850cb99640 1 -- 172.21.15.46:0/1938318031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f802cc20 con 0x7f85080b74f0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.692+0000 7f850cb99640 1 -- 172.21.15.46:0/1938318031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f80343f0 con 0x7f85080b74f0 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 -- 172.21.15.46:0/1938318031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080b74f0 msgr2=0x7f85080b7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080b74f0 0x7f85080b7980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009e50 tx=0x7f84f802bee0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 -- 172.21.15.46:0/1938318031 shutdown_connections 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080b74f0 0x7f85080b7980 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1938318031 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85080d1310 0x7f85080d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 -- 172.21.15.46:0/1938318031 >> 172.21.15.46:0/1938318031 conn(0x7f8508067b40 msgr2=0x7f8508067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 -- 172.21.15.46:0/1938318031 shutdown_connections 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.693+0000 7f850eb9d640 1 -- 172.21.15.46:0/1938318031 wait complete. 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.694+0000 7f850eb9d640 1 Processor -- start 2024-04-15T15:15:39.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.694+0000 7f850eb9d640 1 -- start start 2024-04-15T15:15:39.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850eb9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85080b74f0 0x7f850815ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850eb9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 0x7f85080cd140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850eb9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f850815e760 0x7f8508154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850eb9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85080d3b10 con 0x7f85080d1310 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850eb9d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f85080d3830 con 0x7f85080b74f0 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850eb9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f85080d4030 con 0x7f850815e760 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 0x7f85080cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 0x7f85080cd140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43780/0 (socket says 172.21.15.46:43780) 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 -- 172.21.15.46:0/1152191356 learned_addr learned my addr 172.21.15.46:0/1152191356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850e39c640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f850815e760 0x7f8508154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850db9b640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85080b74f0 0x7f850815ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 -- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f850815e760 msgr2=0x7f8508154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f850815e760 0x7f8508154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 -- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85080b74f0 msgr2=0x7f850815ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85080b74f0 0x7f850815ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850d39a640 1 -- 172.21.15.46:0/1152191356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8508154950 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.695+0000 7f850e39c640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f850815e760 0x7f8508154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.696+0000 7f850d39a640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 0x7f85080cd140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c7f0 tx=0x7f84fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.696+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc015070 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.696+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84fc0079d0 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.696+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850815f410 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.696+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc007da0 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.696+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850815f9d0 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.697+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850804fdc0 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.697+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f84fc018070 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.698+0000 7f85067fc640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f84f406c8f0 0x7f84f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.698+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f84fc08bb10 con 0x7f85080d1310 2024-04-15T15:15:39.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.698+0000 7f850db9b640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f84f406c8f0 0x7f84f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.698+0000 7f850db9b640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f84f406c8f0 0x7f84f406edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f850000a750 tx=0x7f8500008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.702+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84fc059ea0 con 0x7f85080d1310 2024-04-15T15:15:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:39.887+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f85080c7c90 con 0x7f85080d1310 2024-04-15T15:15:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.564+0000 7f85067fc640 1 -- 172.21.15.46:0/1152191356 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f84fc0597a0 con 0x7f85080d1310 2024-04-15T15:15:41.566 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' created 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f84f406c8f0 msgr2=0x7f84f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f84f406c8f0 0x7f84f406edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f850000a750 tx=0x7f8500008040 comp rx=0 tx=0).stop 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 msgr2=0x7f85080cd140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 0x7f85080cd140 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c7f0 tx=0x7f84fc00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 shutdown_connections 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f84f406c8f0 0x7f84f406edb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f850815e760 0x7f8508154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85080d1310 0x7f85080cd140 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 --2- 172.21.15.46:0/1152191356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85080b74f0 0x7f850815ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 >> 172.21.15.46:0/1152191356 conn(0x7f8508067b40 msgr2=0x7f85080b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 shutdown_connections 2024-04-15T15:15:41.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.568+0000 7f850eb9d640 1 -- 172.21.15.46:0/1152191356 wait complete. 2024-04-15T15:15:41.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-15T15:15:41.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.745+0000 7fbc4097c640 1 Processor -- start 2024-04-15T15:15:41.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.745+0000 7fbc4097c640 1 -- start start 2024-04-15T15:15:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc4097c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc3c15eb70 0x7fbc3c0d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc4097c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbc3c0cfe80 0x7fbc3c06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc4097c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc3c0d2c00 con 0x7fbc3c0d1070 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc4097c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbc3c0d2ee0 con 0x7fbc3c15eb70 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc4097c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fbc3c0d31c0 con 0x7fbc3c0cfe80 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc3c15eb70 0x7fbc3c0d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbc3c15eb70 0x7fbc3c0d39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43792/0 (socket says 172.21.15.46:43792) 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 -- 172.21.15.46:0/3783636949 learned_addr learned my addr 172.21.15.46:0/3783636949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc39d9b640 1 --2- 172.21.15.46:0/3783636949 >> v2:172.21.15.78:3300/0 conn(0x7fbc3c0cfe80 0x7fbc3c06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 -- 172.21.15.46:0/3783636949 >> v2:172.21.15.78:3300/0 conn(0x7fbc3c0cfe80 msgr2=0x7fbc3c06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 --2- 172.21.15.46:0/3783636949 >> v2:172.21.15.78:3300/0 conn(0x7fbc3c0cfe80 0x7fbc3c06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 -- 172.21.15.46:0/3783636949 >> v1:172.21.15.78:6789/0 conn(0x7fbc3c0d1070 legacy=0x7fbc3c0b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc3a59c640 1 -- 172.21.15.46:0/3783636949 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3c0d34a0 con 0x7fbc3c15eb70 2024-04-15T15:15:41.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.746+0000 7fbc39d9b640 1 --2- 172.21.15.46:0/3783636949 >> v2:172.21.15.78:3300/0 conn(0x7fbc3c0cfe80 0x7fbc3c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.747+0000 7fbc3a59c640 1 --2- 172.21.15.46:0/3783636949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c15eb70 0x7fbc3c0d39c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c08cb60 tx=0x7fbc3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5130061d0b3ad61 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.747+0000 7fbc3959a640 1 -- 172.21.15.46:0/3783636949 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc34035020 con 0x7fbc3c15eb70 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.747+0000 7fbc3959a640 1 -- 172.21.15.46:0/3783636949 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc34031870 con 0x7fbc3c15eb70 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.747+0000 7fbc3959a640 1 -- 172.21.15.46:0/3783636949 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc34031c40 con 0x7fbc3c15eb70 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 -- 172.21.15.46:0/3783636949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c15eb70 msgr2=0x7fbc3c0d39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 --2- 172.21.15.46:0/3783636949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c15eb70 0x7fbc3c0d39c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c08cb60 tx=0x7fbc3402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 -- 172.21.15.46:0/3783636949 shutdown_connections 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 --2- 172.21.15.46:0/3783636949 >> v2:172.21.15.78:3300/0 conn(0x7fbc3c0cfe80 0x7fbc3c06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 --2- 172.21.15.46:0/3783636949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c15eb70 0x7fbc3c0d39c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 -- 172.21.15.46:0/3783636949 >> 172.21.15.46:0/3783636949 conn(0x7fbc3c0b61d0 msgr2=0x7fbc3c0b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 -- 172.21.15.46:0/3783636949 shutdown_connections 2024-04-15T15:15:41.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.748+0000 7fbc4097c640 1 -- 172.21.15.46:0/3783636949 wait complete. 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.749+0000 7fbc4097c640 1 Processor -- start 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.749+0000 7fbc4097c640 1 -- start start 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc4097c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3c0cfe80 0x7fbc3c0cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc4097c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 0x7fbc3c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc4097c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc3c15eb70 0x7fbc3c0cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc4097c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c06d770 con 0x7fbc3c0d1070 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc4097c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3c06d5f0 con 0x7fbc3c0cfe80 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc4097c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3c0d14e0 con 0x7fbc3c15eb70 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc39d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 0x7fbc3c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc39d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 0x7fbc3c0ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43798/0 (socket says 172.21.15.46:43798) 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc39d9b640 1 -- 172.21.15.46:0/4071105480 learned_addr learned my addr 172.21.15.46:0/4071105480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:41.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.750+0000 7fbc3a59c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3c0cfe80 0x7fbc3c0cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc3ad9d640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc3c15eb70 0x7fbc3c0cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc39d9b640 1 -- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc3c15eb70 msgr2=0x7fbc3c0cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc39d9b640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc3c15eb70 0x7fbc3c0cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc39d9b640 1 -- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3c0cfe80 msgr2=0x7fbc3c0cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc39d9b640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3c0cfe80 0x7fbc3c0cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc39d9b640 1 -- 172.21.15.46:0/4071105480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3c0cfc90 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc39d9b640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 0x7fbc3c0ce550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400c7f0 tx=0x7fbc2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24015070 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc24007a30 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc3c0cf0e0 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24007e00 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.751+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc3c0cf920 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.752+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc3c0689e0 con 0x7fbc3c0d1070 2024-04-15T15:15:41.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.753+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc24018070 con 0x7fbc3c0d1070 2024-04-15T15:15:41.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.753+0000 7fbc32ffd640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.753+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fbc2408bda0 con 0x7fbc3c0d1070 2024-04-15T15:15:41.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.753+0000 7fbc3a59c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.754+0000 7fbc3a59c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbc34036000 tx=0x7fbc3403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.758+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc3c0cf0e0 con 0x7fbc3c0d1070 2024-04-15T15:15:41.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:41.940+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fbc3c1580a0 con 0x7fbc3c0d1070 2024-04-15T15:15:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.566+0000 7fbc32ffd640 1 -- 172.21.15.46:0/4071105480 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7fbc24059f40 con 0x7fbc3c0d1070 2024-04-15T15:15:42.570 INFO:teuthology.orchestra.run.smithi046.stderr:set pool 4 allow_ec_overwrites to true 2024-04-15T15:15:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.569+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1c06c8f0 msgr2=0x7fbc1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.569+0000 7fbc4097c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbc34036000 tx=0x7fbc3403c040 comp rx=0 tx=0).stop 2024-04-15T15:15:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.569+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 msgr2=0x7fbc3c0ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.569+0000 7fbc4097c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 0x7fbc3c0ce550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400c7f0 tx=0x7fbc2400cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 shutdown_connections 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1c06c8f0 0x7fbc1c06edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc3c15eb70 0x7fbc3c0cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc3c0d1070 0x7fbc3c0ce550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 --2- 172.21.15.46:0/4071105480 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc3c0cfe80 0x7fbc3c0cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 >> 172.21.15.46:0/4071105480 conn(0x7fbc3c0b61d0 msgr2=0x7fbc3c0b5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 shutdown_connections 2024-04-15T15:15:42.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.570+0000 7fbc4097c640 1 -- 172.21.15.46:0/4071105480 wait complete. 2024-04-15T15:15:42.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-15T15:15:42.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.716+0000 7f9c2344e640 1 Processor -- start 2024-04-15T15:15:42.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 -- start start 2024-04-15T15:15:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c1c15eb00 0x7f9c1c0ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9c1c0b0560 0x7f9c1c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b09a0 0x7f9c1c06bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9c1c15eee0 con 0x7f9c1c15eb00 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f9c1c0cbd10 con 0x7f9c1c0b0560 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c2344e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1c0cbff0 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.717+0000 7f9c211ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c1c15eb00 0x7f9c1c0ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c1c15eb00 0x7f9c1c0ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43804/0 (socket says 172.21.15.46:43804) 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 -- 172.21.15.46:0/2441019322 learned_addr learned my addr 172.21.15.46:0/2441019322 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c219eb640 1 --2- 172.21.15.46:0/2441019322 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b09a0 0x7f9c1c06bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c209e9640 1 --2- 172.21.15.46:0/2441019322 >> v2:172.21.15.78:3300/0 conn(0x7f9c1c0b0560 0x7f9c1c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 -- 172.21.15.46:0/2441019322 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b09a0 msgr2=0x7f9c1c06bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 --2- 172.21.15.46:0/2441019322 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b09a0 0x7f9c1c06bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 -- 172.21.15.46:0/2441019322 >> v2:172.21.15.78:3300/0 conn(0x7f9c1c0b0560 msgr2=0x7f9c1c06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 --2- 172.21.15.46:0/2441019322 >> v2:172.21.15.78:3300/0 conn(0x7f9c1c0b0560 0x7f9c1c06b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.718+0000 7f9c211ea640 1 -- 172.21.15.46:0/2441019322 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1c0cc2d0 con 0x7f9c1c15eb00 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c211ea640 1 --2- 172.21.15.46:0/2441019322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c0ae2a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0cf0b0 tx=0x7f9c1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de778c3afb542ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c1bfff640 1 -- 172.21.15.46:0/2441019322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14035020 con 0x7f9c1c15eb00 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c1bfff640 1 -- 172.21.15.46:0/2441019322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c14031870 con 0x7f9c1c15eb00 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c1bfff640 1 -- 172.21.15.46:0/2441019322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14031c40 con 0x7f9c1c15eb00 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 -- 172.21.15.46:0/2441019322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c15eb00 msgr2=0x7f9c1c0ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 --2- 172.21.15.46:0/2441019322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c0ae2a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0cf0b0 tx=0x7f9c1402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 -- 172.21.15.46:0/2441019322 shutdown_connections 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 --2- 172.21.15.46:0/2441019322 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b09a0 0x7f9c1c06bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 --2- 172.21.15.46:0/2441019322 >> v2:172.21.15.78:3300/0 conn(0x7f9c1c0b0560 0x7f9c1c06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 --2- 172.21.15.46:0/2441019322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c0ae2a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 -- 172.21.15.46:0/2441019322 >> 172.21.15.46:0/2441019322 conn(0x7f9c1c09a9d0 msgr2=0x7f9c1c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 -- 172.21.15.46:0/2441019322 shutdown_connections 2024-04-15T15:15:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.719+0000 7f9c2344e640 1 -- 172.21.15.46:0/2441019322 wait complete. 2024-04-15T15:15:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 Processor -- start 2024-04-15T15:15:42.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 -- start start 2024-04-15T15:15:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b0560 0x7f9c1c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 0x7f9c1c1f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c1f8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c0ae7e0 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1c0cc890 con 0x7f9c1c15eb00 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c2344e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9c1c0cec60 con 0x7f9c1c0b0560 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c209e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 0x7f9c1c1f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c209e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 0x7f9c1c1f7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43812/0 (socket says 172.21.15.46:43812) 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.721+0000 7f9c209e9640 1 -- 172.21.15.46:0/3662827678 learned_addr learned my addr 172.21.15.46:0/3662827678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c211ea640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b0560 0x7f9c1c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c219eb640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c1f8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c209e9640 1 -- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b0560 msgr2=0x7f9c1c157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c209e9640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b0560 0x7f9c1c157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c209e9640 1 -- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c1c15eb00 msgr2=0x7f9c1c1f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c209e9640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c1f8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c209e9640 1 -- 172.21.15.46:0/3662827678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1c0c8800 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c209e9640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 0x7f9c1c1f7d10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000e8c0 tx=0x7f9c1000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c10017070 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c1000c940 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1c0b5a30 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000cd10 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.722+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c1c157950 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.723+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c1c0690f0 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.724+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c1001a070 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.724+0000 7f9c19ffb640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9c0806c8f0 0x7f9c0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.724+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f9c1008bd20 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.725+0000 7f9c211ea640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9c0806c8f0 0x7f9c0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.725+0000 7f9c211ea640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9c0806c8f0 0x7f9c0806edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9c14036000 tx=0x7f9c1403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.729+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c10059870 con 0x7f9c1c0b09a0 2024-04-15T15:15:42.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:42.954+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f9c1c0b33d0 con 0x7f9c1c0b09a0 2024-04-15T15:15:43.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.583+0000 7f9c19ffb640 1 -- 172.21.15.46:0/3662827678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7f9c1005db70 con 0x7f9c1c0b09a0 2024-04-15T15:15:43.585 INFO:teuthology.orchestra.run.smithi046.stderr:added data pool 4 to fsmap 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9c0806c8f0 msgr2=0x7f9c0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9c0806c8f0 0x7f9c0806edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9c14036000 tx=0x7f9c1403c040 comp rx=0 tx=0).stop 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 msgr2=0x7f9c1c1f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 0x7f9c1c1f7d10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000e8c0 tx=0x7f9c1000ed80 comp rx=0 tx=0).stop 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 shutdown_connections 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9c0806c8f0 0x7f9c0806edb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c1c15eb00 0x7f9c1c1f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c1c0b09a0 0x7f9c1c1f7d10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 --2- 172.21.15.46:0/3662827678 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9c1c0b0560 0x7f9c1c157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 >> 172.21.15.46:0/3662827678 conn(0x7f9c1c09a9d0 msgr2=0x7f9c1c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 shutdown_connections 2024-04-15T15:15:43.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.587+0000 7f9c2344e640 1 -- 172.21.15.46:0/3662827678 wait complete. 2024-04-15T15:15:43.602 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.711+0000 7fa6a4940640 1 Processor -- start 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.711+0000 7fa6a4940640 1 -- start start 2024-04-15T15:15:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa6a4940640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa6a4940640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6a00d1440 0x7fa6a00b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa6a4940640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6a00d2480 con 0x7fa6a015e790 2024-04-15T15:15:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa6a4940640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa6a00d2db0 con 0x7fa6a00d1440 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa6a4940640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a00d2760 con 0x7fa6a00d3240 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa69e59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6a015e790 0x7fa6a00cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49754/0 (socket says 172.21.15.46:49754) 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa69e59c640 1 -- 172.21.15.46:0/2443539234 learned_addr learned my addr 172.21.15.46:0/2443539234 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa697fff640 1 --2- 172.21.15.46:0/2443539234 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/2443539234 >> v2:172.21.15.78:3300/0 conn(0x7fa6a00d1440 0x7fa6a00b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa69dd9b640 1 -- 172.21.15.46:0/2443539234 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1839898828 0 0) 0x7fa6a00d2480 con 0x7fa6a015e790 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa69dd9b640 1 -- 172.21.15.46:0/2443539234 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa68c003520 con 0x7fa6a015e790 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa697fff640 1 -- 172.21.15.46:0/2443539234 >> v2:172.21.15.78:3300/0 conn(0x7fa6a00d1440 msgr2=0x7fa6a00b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa697fff640 1 --2- 172.21.15.46:0/2443539234 >> v2:172.21.15.78:3300/0 conn(0x7fa6a00d1440 0x7fa6a00b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa697fff640 1 -- 172.21.15.46:0/2443539234 >> v1:172.21.15.46:6789/0 conn(0x7fa6a015e790 legacy=0x7fa6a00cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.712+0000 7fa697fff640 1 -- 172.21.15.46:0/2443539234 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a00d2a40 con 0x7fa6a00d3240 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.713+0000 7fa697fff640 1 --2- 172.21.15.46:0/2443539234 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00d0f00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa6880097a0 tx=0x7fa68802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1053838174a4f14 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.713+0000 7fa69dd9b640 1 -- 172.21.15.46:0/2443539234 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688035020 con 0x7fa6a00d3240 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.713+0000 7fa69dd9b640 1 -- 172.21.15.46:0/2443539234 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa688031830 con 0x7fa6a00d3240 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.713+0000 7fa69dd9b640 1 -- 172.21.15.46:0/2443539234 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa688031c00 con 0x7fa6a00d3240 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 -- 172.21.15.46:0/2443539234 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 msgr2=0x7fa6a00d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 --2- 172.21.15.46:0/2443539234 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00d0f00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa6880097a0 tx=0x7fa68802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 -- 172.21.15.46:0/2443539234 shutdown_connections 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 --2- 172.21.15.46:0/2443539234 >> v2:172.21.15.78:3300/0 conn(0x7fa6a00d1440 0x7fa6a00b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 --2- 172.21.15.46:0/2443539234 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00d0f00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 -- 172.21.15.46:0/2443539234 >> 172.21.15.46:0/2443539234 conn(0x7fa6a008bf60 msgr2=0x7fa6a00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 -- 172.21.15.46:0/2443539234 shutdown_connections 2024-04-15T15:15:43.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.714+0000 7fa6a4940640 1 -- 172.21.15.46:0/2443539234 wait complete. 2024-04-15T15:15:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.715+0000 7fa6a4940640 1 Processor -- start 2024-04-15T15:15:43.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.715+0000 7fa6a4940640 1 -- start start 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa6a4940640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6a00d1440 0x7fa6a014bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa6a4940640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00b97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa6a4940640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 0x7fa6a00b9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa6a4940640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a00d5180 con 0x7fa6a015e790 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa6a4940640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a00d0430 con 0x7fa6a00d1440 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa6a4940640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a00d5300 con 0x7fa6a00d3240 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 0x7fa6a00b9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 0x7fa6a00b9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43820/0 (socket says 172.21.15.46:43820) 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 -- 172.21.15.46:0/4012991658 learned_addr learned my addr 172.21.15.46:0/4012991658 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69e59c640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6a00d1440 0x7fa6a014bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa697fff640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00b97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 -- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 msgr2=0x7fa6a00b97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00b97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 -- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6a00d1440 msgr2=0x7fa6a014bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6a00d1440 0x7fa6a014bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 -- 172.21.15.46:0/4012991658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a01ea3b0 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69e59c640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6a00d1440 0x7fa6a014bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa697fff640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00b97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.716+0000 7fa69ed9d640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 0x7fa6a00b9cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa698009dd0 tx=0x7fa69800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.717+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698018020 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.717+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa698014480 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.717+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a015a0a0 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.717+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698014870 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.717+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a015a660 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.718+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a00689e0 con 0x7fa6a015e790 2024-04-15T15:15:43.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.718+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa698005d10 con 0x7fa6a015e790 2024-04-15T15:15:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.719+0000 7fa696ffd640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa68006c8f0 0x7fa68006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.719+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa69808c970 con 0x7fa6a015e790 2024-04-15T15:15:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.719+0000 7fa69e59c640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa68006c8f0 0x7fa68006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.719+0000 7fa69e59c640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa68006c8f0 0x7fa68006edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa690007fd0 tx=0x7fa690009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.723+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa698022070 con 0x7fa6a015e790 2024-04-15T15:15:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.950+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6a00b7460 con 0x7fa6a015e790 2024-04-15T15:15:43.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.950+0000 7fa696ffd640 1 -- 172.21.15.46:0/4012991658 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5174 (secure 0 0 0) 0x7fa698010040 con 0x7fa6a015e790 2024-04-15T15:15:43.952 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:43.952 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:15:42.956426+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:15:43.953 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa68006c8f0 msgr2=0x7fa68006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa68006c8f0 0x7fa68006edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa690007fd0 tx=0x7fa690009040 comp rx=0 tx=0).stop 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 msgr2=0x7fa6a00b9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 0x7fa6a00b9cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa698009dd0 tx=0x7fa69800c490 comp rx=0 tx=0).stop 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 shutdown_connections 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa68006c8f0 0x7fa68006edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6a015e790 0x7fa6a00b9cf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6a00d3240 0x7fa6a00b97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 --2- 172.21.15.46:0/4012991658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6a00d1440 0x7fa6a014bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 >> 172.21.15.46:0/4012991658 conn(0x7fa6a008bf60 msgr2=0x7fa6a00d4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 shutdown_connections 2024-04-15T15:15:43.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:43.953+0000 7fa6a4940640 1 -- 172.21.15.46:0/4012991658 wait complete. 2024-04-15T15:15:43.966 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 Processor -- start 2024-04-15T15:15:44.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 -- start start 2024-04-15T15:15:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f376c15ebb0 0x7f376c06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 0x7f376c0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c0cccb0 con 0x7f376c09a9d0 2024-04-15T15:15:44.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f376c0cd8c0 con 0x7f376c15ebb0 2024-04-15T15:15:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.069+0000 7f3772352640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f376c0ced60 con 0x7f376c09b780 2024-04-15T15:15:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 0x7f376c0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f3763fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f376c15ebb0 0x7f376c06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 0x7f376c0b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:38336/0 (socket says 172.21.15.46:38336) 2024-04-15T15:15:44.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 -- 172.21.15.46:0/3960490940 learned_addr learned my addr 172.21.15.46:0/3960490940 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:44.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 -- 172.21.15.46:0/3960490940 >> v2:172.21.15.78:3300/0 conn(0x7f376c15ebb0 msgr2=0x7f376c06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 --2- 172.21.15.46:0/3960490940 >> v2:172.21.15.78:3300/0 conn(0x7f376c15ebb0 0x7f376c06cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 -- 172.21.15.46:0/3960490940 >> v1:172.21.15.78:6789/0 conn(0x7f376c09a9d0 legacy=0x7f376c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f37708ef640 1 -- 172.21.15.46:0/3960490940 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c0ccf90 con 0x7f376c09b780 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.070+0000 7f3763fff640 1 --2- 172.21.15.46:0/3960490940 >> v2:172.21.15.78:3300/0 conn(0x7f376c15ebb0 0x7f376c06cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.071+0000 7f37708ef640 1 --2- 172.21.15.46:0/3960490940 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 0x7f376c0b76f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3764009e50 tx=0x7f376402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd46b3f0a487ca21 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.071+0000 7f376b7fe640 1 -- 172.21.15.46:0/3960490940 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376402cb00 con 0x7f376c09b780 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.071+0000 7f376b7fe640 1 -- 172.21.15.46:0/3960490940 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f376402cca0 con 0x7f376c09b780 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.071+0000 7f376b7fe640 1 -- 172.21.15.46:0/3960490940 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764034610 con 0x7f376c09b780 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 -- 172.21.15.46:0/3960490940 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 msgr2=0x7f376c0b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 --2- 172.21.15.46:0/3960490940 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 0x7f376c0b76f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3764009e50 tx=0x7f376402bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 -- 172.21.15.46:0/3960490940 shutdown_connections 2024-04-15T15:15:44.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 --2- 172.21.15.46:0/3960490940 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c09b780 0x7f376c0b76f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 --2- 172.21.15.46:0/3960490940 >> v2:172.21.15.78:3300/0 conn(0x7f376c15ebb0 0x7f376c06cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 -- 172.21.15.46:0/3960490940 >> 172.21.15.46:0/3960490940 conn(0x7f376c0b3fb0 msgr2=0x7f376c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 -- 172.21.15.46:0/3960490940 shutdown_connections 2024-04-15T15:15:44.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.072+0000 7f3772352640 1 -- 172.21.15.46:0/3960490940 wait complete. 2024-04-15T15:15:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.073+0000 7f3772352640 1 Processor -- start 2024-04-15T15:15:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.073+0000 7f3772352640 1 -- start start 2024-04-15T15:15:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 0x7f376c0c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f376c09b780 0x7f376c15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c157080 0x7f376c0cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06d240 con 0x7f376c09a9d0 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f376c0cd550 con 0x7f376c09b780 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f376c0b5310 con 0x7f376c157080 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 0x7f376c0c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 0x7f376c0c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43830/0 (socket says 172.21.15.46:43830) 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 -- 172.21.15.46:0/3226366642 learned_addr learned my addr 172.21.15.46:0/3226366642 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f37708ef640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c157080 0x7f376c0cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3763fff640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f376c09b780 0x7f376c15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 -- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c157080 msgr2=0x7f376c0cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c157080 0x7f376c0cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 -- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f376c09b780 msgr2=0x7f376c15efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f376c09b780 0x7f376c15efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 -- 172.21.15.46:0/3226366642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c15ff90 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f376bfff640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 0x7f376c0c5ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c400 tx=0x7f375c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c0052d0 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f375c005490 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c15f6f0 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c007260 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.074+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c15fcb0 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.075+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c0690f0 con 0x7f376c09a9d0 2024-04-15T15:15:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.076+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f375c007400 con 0x7f376c09a9d0 2024-04-15T15:15:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.076+0000 7f37697fa640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f374c06c8f0 0x7f374c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.076+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f375c08b410 con 0x7f376c09a9d0 2024-04-15T15:15:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.076+0000 7f3763fff640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f374c06c8f0 0x7f374c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.077+0000 7f3763fff640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f374c06c8f0 0x7f374c06edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3754009560 tx=0x7f3754009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.081+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f375c058e10 con 0x7f376c09a9d0 2024-04-15T15:15:44.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.266+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f376c067a30 con 0x7f376c09a9d0 2024-04-15T15:15:44.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.267+0000 7f37697fa640 1 -- 172.21.15.46:0/3226366642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f375c05d110 con 0x7f376c09a9d0 2024-04-15T15:15:44.269 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:44.269 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f374c06c8f0 msgr2=0x7f374c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f374c06c8f0 0x7f374c06edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3754009560 tx=0x7f3754009190 comp rx=0 tx=0).stop 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 msgr2=0x7f376c0c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 0x7f376c0c5ff0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f375c00c400 tx=0x7f375c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 shutdown_connections 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f374c06c8f0 0x7f374c06edb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f376c157080 0x7f376c0cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f376c09b780 0x7f376c15efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 --2- 172.21.15.46:0/3226366642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f376c09a9d0 0x7f376c0c5ff0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 >> 172.21.15.46:0/3226366642 conn(0x7f376c0b3fb0 msgr2=0x7f376c0b75d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 shutdown_connections 2024-04-15T15:15:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.269+0000 7f3772352640 1 -- 172.21.15.46:0/3226366642 wait complete. 2024-04-15T15:15:44.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.385+0000 7fce2fa77640 1 Processor -- start 2024-04-15T15:15:44.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.385+0000 7fce2fa77640 1 -- start start 2024-04-15T15:15:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2fa77640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fce2809a9d0 0x7fce2809adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2fa77640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce2815ebb0 0x7fce2806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2fa77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce2809b2f0 con 0x7fce2809b780 2024-04-15T15:15:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2fa77640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fce280cd8c0 con 0x7fce2815ebb0 2024-04-15T15:15:44.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2fa77640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fce280ced60 con 0x7fce2809a9d0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2d012640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce2815ebb0 0x7fce2806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2d012640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fce2815ebb0 0x7fce2806cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43838/0 (socket says 172.21.15.46:43838) 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.386+0000 7fce2d012640 1 -- 172.21.15.46:0/3749841519 learned_addr learned my addr 172.21.15.46:0/3749841519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d813640 1 --2- 172.21.15.46:0/3749841519 >> v2:172.21.15.78:3300/0 conn(0x7fce2809a9d0 0x7fce2809adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d012640 1 -- 172.21.15.46:0/3749841519 >> v2:172.21.15.78:3300/0 conn(0x7fce2809a9d0 msgr2=0x7fce2809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d012640 1 --2- 172.21.15.46:0/3749841519 >> v2:172.21.15.78:3300/0 conn(0x7fce2809a9d0 0x7fce2809adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d012640 1 -- 172.21.15.46:0/3749841519 >> v1:172.21.15.78:6789/0 conn(0x7fce2809b780 legacy=0x7fce280b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d012640 1 -- 172.21.15.46:0/3749841519 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce280cccb0 con 0x7fce2815ebb0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d813640 1 --2- 172.21.15.46:0/3749841519 >> v2:172.21.15.78:3300/0 conn(0x7fce2809a9d0 0x7fce2809adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.387+0000 7fce2d012640 1 --2- 172.21.15.46:0/3749841519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2815ebb0 0x7fce2806cd00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce28072550 tx=0x7fce1c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c2e65dc63c7f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2c811640 1 -- 172.21.15.46:0/3749841519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c034070 con 0x7fce2815ebb0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2c811640 1 -- 172.21.15.46:0/3749841519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1c030860 con 0x7fce2815ebb0 2024-04-15T15:15:44.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2c811640 1 -- 172.21.15.46:0/3749841519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c030c30 con 0x7fce2815ebb0 2024-04-15T15:15:44.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 -- 172.21.15.46:0/3749841519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2815ebb0 msgr2=0x7fce2806cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 --2- 172.21.15.46:0/3749841519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2815ebb0 0x7fce2806cd00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fce28072550 tx=0x7fce1c02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 -- 172.21.15.46:0/3749841519 shutdown_connections 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 --2- 172.21.15.46:0/3749841519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2815ebb0 0x7fce2806cd00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 --2- 172.21.15.46:0/3749841519 >> v2:172.21.15.78:3300/0 conn(0x7fce2809a9d0 0x7fce2809adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 -- 172.21.15.46:0/3749841519 >> 172.21.15.46:0/3749841519 conn(0x7fce280b3fb0 msgr2=0x7fce280b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 -- 172.21.15.46:0/3749841519 shutdown_connections 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.388+0000 7fce2fa77640 1 -- 172.21.15.46:0/3749841519 wait complete. 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.389+0000 7fce2fa77640 1 Processor -- start 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.389+0000 7fce2fa77640 1 -- start start 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2fa77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 0x7fce28158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2fa77640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fce2809b780 0x7fce280c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2fa77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce281f5210 0x7fce280c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2fa77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fce280cd550 con 0x7fce2809a9d0 2024-04-15T15:15:44.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2fa77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fce280cd270 con 0x7fce281f5210 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2fa77640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fce2806d240 con 0x7fce2809b780 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2d813640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 0x7fce28158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2d813640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 0x7fce28158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43852/0 (socket says 172.21.15.46:43852) 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2d813640 1 -- 172.21.15.46:0/1836250037 learned_addr learned my addr 172.21.15.46:0/1836250037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d012640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fce2809b780 0x7fce280c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.390+0000 7fce2e014640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce281f5210 0x7fce280c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d813640 1 -- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fce2809b780 msgr2=0x7fce280c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d813640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fce2809b780 0x7fce280c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d813640 1 -- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce281f5210 msgr2=0x7fce280c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d813640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce281f5210 0x7fce280c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d813640 1 -- 172.21.15.46:0/1836250037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce281f4490 con 0x7fce2809a9d0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2d813640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 0x7fce28158450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce2000e730 tx=0x7fce2000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2000f040 con 0x7fce2809a9d0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce20004230 con 0x7fce2809a9d0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce281f4730 con 0x7fce2809a9d0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2000ee40 con 0x7fce2809a9d0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.391+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce281f4cf0 con 0x7fce2809a9d0 2024-04-15T15:15:44.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.392+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce280690f0 con 0x7fce2809a9d0 2024-04-15T15:15:44.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.392+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fce2000c6d0 con 0x7fce2809a9d0 2024-04-15T15:15:44.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.393+0000 7fce267fc640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fce1406c8f0 0x7fce1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.393+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fce2008bc40 con 0x7fce2809a9d0 2024-04-15T15:15:44.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.393+0000 7fce2d012640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fce1406c8f0 0x7fce1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.394+0000 7fce2d012640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fce1406c8f0 0x7fce1406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce1c036000 tx=0x7fce1c03b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.397+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce20017020 con 0x7fce2809a9d0 2024-04-15T15:15:44.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.587+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce28161db0 con 0x7fce2809a9d0 2024-04-15T15:15:44.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.588+0000 7fce267fc640 1 -- 172.21.15.46:0/1836250037 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fce20059d30 con 0x7fce2809a9d0 2024-04-15T15:15:44.590 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:44.590 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fce1406c8f0 msgr2=0x7fce1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fce1406c8f0 0x7fce1406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fce1c036000 tx=0x7fce1c03b040 comp rx=0 tx=0).stop 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 msgr2=0x7fce28158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 0x7fce28158450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fce2000e730 tx=0x7fce2000ebf0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 shutdown_connections 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fce1406c8f0 0x7fce1406edb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce281f5210 0x7fce280c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fce2809b780 0x7fce280c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 --2- 172.21.15.46:0/1836250037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce2809a9d0 0x7fce28158450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 >> 172.21.15.46:0/1836250037 conn(0x7fce280b3fb0 msgr2=0x7fce2806fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 shutdown_connections 2024-04-15T15:15:44.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.591+0000 7fce2fa77640 1 -- 172.21.15.46:0/1836250037 wait complete. 2024-04-15T15:15:44.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.727+0000 7fa83d731640 1 Processor -- start 2024-04-15T15:15:44.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.727+0000 7fa83d731640 1 -- start start 2024-04-15T15:15:44.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa83d731640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa8380d3270 0x7fa8380cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa83d731640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8380b47a0 con 0x7fa8380d2360 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa83d731640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8380b4a80 con 0x7fa83815e700 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa83d731640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa8380b53b0 con 0x7fa8380d3270 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa8367fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa8380d2360 0x7fa8380d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49770/0 (socket says 172.21.15.46:49770) 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa8367fc640 1 -- 172.21.15.46:0/3745119584 learned_addr learned my addr 172.21.15.46:0/3745119584 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1954837455 0 0) 0x7fa8380b47a0 con 0x7fa8380d2360 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa824003520 con 0x7fa8380d2360 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.728+0000 7fa8377fe640 1 --2- 172.21.15.46:0/3745119584 >> v2:172.21.15.78:3300/0 conn(0x7fa8380d3270 0x7fa8380cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 947657118 0 0) 0x7fa824003520 con 0x7fa8380d2360 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 >> v2:172.21.15.78:3300/0 conn(0x7fa8380d3270 msgr2=0x7fa8380cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 --2- 172.21.15.46:0/3745119584 >> v2:172.21.15.78:3300/0 conn(0x7fa8380d3270 0x7fa8380cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 >> v1:172.21.15.78:6789/0 conn(0x7fa83815e700 legacy=0x7fa8380b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8380b4d60 con 0x7fa8380d2360 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fa820001670 con 0x7fa8380d2360 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 >> v1:172.21.15.46:6789/0 conn(0x7fa8380d2360 legacy=0x7fa8380d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8240039b0 0x7fa824003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa824004800 0x7fa824004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8240051c0 0x7fa82400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa8380b53b0 con 0x7fa8240039b0 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa82400c090 con 0x7fa8240051c0 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa82400c290 con 0x7fa824004800 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa836ffd640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8240039b0 0x7fa824003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa8377fe640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8240051c0 0x7fa82400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa8367fc640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa824004800 0x7fa824004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa836ffd640 1 -- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa824004800 msgr2=0x7fa824004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa836ffd640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa824004800 0x7fa824004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa836ffd640 1 -- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8240051c0 msgr2=0x7fa82400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa836ffd640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8240051c0 0x7fa82400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa836ffd640 1 -- 172.21.15.46:0/3745119584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa824003520 con 0x7fa8240039b0 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa8377fe640 1 -- 172.21.15.46:0/3745119584 reap_dead start 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.729+0000 7fa8367fc640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa824004800 0x7fa824004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa836ffd640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8240039b0 0x7fa824003df0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa828001e70 tx=0x7fa824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c6e09ec87059fa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828001ac0 con 0x7fa8240039b0 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa828003bc0 con 0x7fa8240039b0 2024-04-15T15:15:44.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa835ffb640 1 -- 172.21.15.46:0/3745119584 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8280050d0 con 0x7fa8240039b0 2024-04-15T15:15:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa83d731640 1 -- 172.21.15.46:0/3745119584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8240039b0 msgr2=0x7fa824003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa83d731640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8240039b0 0x7fa824003df0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa828001e70 tx=0x7fa824005830 comp rx=0 tx=0).stop 2024-04-15T15:15:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa83d731640 1 -- 172.21.15.46:0/3745119584 shutdown_connections 2024-04-15T15:15:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa83d731640 1 --2- 172.21.15.46:0/3745119584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8240039b0 0x7fa824003df0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.730+0000 7fa83d731640 1 -- 172.21.15.46:0/3745119584 >> 172.21.15.46:0/3745119584 conn(0x7fa83808bf60 msgr2=0x7fa8380d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.731+0000 7fa83d731640 1 -- 172.21.15.46:0/3745119584 shutdown_connections 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.731+0000 7fa83d731640 1 -- 172.21.15.46:0/3745119584 wait complete. 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.732+0000 7fa83d731640 1 Processor -- start 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.732+0000 7fa83d731640 1 -- start start 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.732+0000 7fa83d731640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8240039b0 0x7fa83814d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8381f8e60 0x7fa83814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 0x7fa8381f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa8380d1d40 con 0x7fa8380c9d50 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa8380b4450 con 0x7fa8381f8e60 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa8380d1ec0 con 0x7fa8240039b0 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 0x7fa8381f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 0x7fa8381f3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43868/0 (socket says 172.21.15.46:43868) 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 -- 172.21.15.46:0/1190530293 learned_addr learned my addr 172.21.15.46:0/1190530293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa836ffd640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8240039b0 0x7fa83814d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8367fc640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8381f8e60 0x7fa83814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 -- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8240039b0 msgr2=0x7fa83814d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8240039b0 0x7fa83814d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 -- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8381f8e60 msgr2=0x7fa83814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8381f8e60 0x7fa83814f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 -- 172.21.15.46:0/1190530293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83814f9c0 con 0x7fa8380c9d50 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8367fc640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8381f8e60 0x7fa83814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa8377fe640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 0x7fa8381f3ff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa83806c640 tx=0x7fa830011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830004240 con 0x7fa8380c9d50 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83001c070 con 0x7fa8380c9d50 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83814fca0 con 0x7fa8380c9d50 2024-04-15T15:15:44.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83000f710 con 0x7fa8380c9d50 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.733+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8380b9060 con 0x7fa8380c9d50 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.734+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8380b9260 con 0x7fa8380c9d50 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.735+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa830012040 con 0x7fa8380c9d50 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.735+0000 7fa82f7fe640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa81806c8f0 0x7fa81806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.735+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa83008bd20 con 0x7fa8380c9d50 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.735+0000 7fa836ffd640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa81806c8f0 0x7fa81806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.736+0000 7fa836ffd640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa81806c8f0 0x7fa81806edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa828001e70 tx=0x7fa82803b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.739+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8380b9260 con 0x7fa8380c9d50 2024-04-15T15:15:44.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.966+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8380cb440 con 0x7fa8380c9d50 2024-04-15T15:15:44.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.966+0000 7fa82f7fe640 1 -- 172.21.15.46:0/1190530293 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5174 (secure 0 0 0) 0x7fa830006bf0 con 0x7fa8380c9d50 2024-04-15T15:15:44.968 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:44.968 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:15:42.956426+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:15:44.969 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2024-04-15T15:15:44.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa81806c8f0 msgr2=0x7fa81806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa81806c8f0 0x7fa81806edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa828001e70 tx=0x7fa82803b040 comp rx=0 tx=0).stop 2024-04-15T15:15:44.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 msgr2=0x7fa8381f3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 0x7fa8381f3ff0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa83806c640 tx=0x7fa830011cf0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 shutdown_connections 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa81806c8f0 0x7fa81806edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8380c9d50 0x7fa8381f3ff0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8381f8e60 0x7fa83814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 --2- 172.21.15.46:0/1190530293 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8240039b0 0x7fa83814d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 >> 172.21.15.46:0/1190530293 conn(0x7fa83808bf60 msgr2=0x7fa8380c7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 shutdown_connections 2024-04-15T15:15:44.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:44.968+0000 7fa83d731640 1 -- 172.21.15.46:0/1190530293 wait complete. 2024-04-15T15:15:44.981 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.084+0000 7f4c842d9640 1 Processor -- start 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.084+0000 7f4c842d9640 1 -- start start 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c842d9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c15e700 0x7f4c7c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c842d9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4c7c0d2360 0x7f4c7c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c842d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c7c15eae0 0x7f4c7c0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c842d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4c7c0b53b0 con 0x7f4c7c15eae0 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c842d9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f4c7c0b47a0 con 0x7f4c7c0d2360 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c842d9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7c0b4a80 con 0x7f4c7c15e700 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c7c15eae0 0x7f4c7c0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4c7c15eae0 0x7f4c7c0d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43878/0 (socket says 172.21.15.46:43878) 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 -- 172.21.15.46:0/3735313126 learned_addr learned my addr 172.21.15.46:0/3735313126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82075640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c15e700 0x7f4c7c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c81874640 1 --2- 172.21.15.46:0/3735313126 >> v2:172.21.15.78:3300/0 conn(0x7f4c7c0d2360 0x7f4c7c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 -- 172.21.15.46:0/3735313126 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c15e700 msgr2=0x7f4c7c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c15e700 0x7f4c7c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 -- 172.21.15.46:0/3735313126 >> v2:172.21.15.78:3300/0 conn(0x7f4c7c0d2360 msgr2=0x7f4c7c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 --2- 172.21.15.46:0/3735313126 >> v2:172.21.15.78:3300/0 conn(0x7f4c7c0d2360 0x7f4c7c0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82876640 1 -- 172.21.15.46:0/3735313126 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c0b4d60 con 0x7f4c7c15eae0 2024-04-15T15:15:45.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.085+0000 7f4c82075640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c15e700 0x7f4c7c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:45.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.086+0000 7f4c82876640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0d30a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0d2100 tx=0x7f4c7402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3e2aaacfd051a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.086+0000 7f4c81073640 1 -- 172.21.15.46:0/3735313126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7402ca70 con 0x7f4c7c15eae0 2024-04-15T15:15:45.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.086+0000 7f4c81073640 1 -- 172.21.15.46:0/3735313126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7402cc10 con 0x7f4c7c15eae0 2024-04-15T15:15:45.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.086+0000 7f4c81073640 1 -- 172.21.15.46:0/3735313126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740343f0 con 0x7f4c7c15eae0 2024-04-15T15:15:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.086+0000 7f4c842d9640 1 -- 172.21.15.46:0/3735313126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 msgr2=0x7f4c7c0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.086+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0d30a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0d2100 tx=0x7f4c7402bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 -- 172.21.15.46:0/3735313126 shutdown_connections 2024-04-15T15:15:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0d30a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3735313126 >> v2:172.21.15.78:3300/0 conn(0x7f4c7c0d2360 0x7f4c7c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3735313126 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c15e700 0x7f4c7c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 -- 172.21.15.46:0/3735313126 >> 172.21.15.46:0/3735313126 conn(0x7f4c7c08bf60 msgr2=0x7f4c7c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 -- 172.21.15.46:0/3735313126 shutdown_connections 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.087+0000 7f4c842d9640 1 -- 172.21.15.46:0/3735313126 wait complete. 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.088+0000 7f4c842d9640 1 Processor -- start 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.088+0000 7f4c842d9640 1 -- start start 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.088+0000 7f4c842d9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c0d2360 0x7f4c7c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.088+0000 7f4c842d9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c7c15e700 0x7f4c7c0b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c842d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c842d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c0b5d80 con 0x7f4c7c15eae0 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c842d9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c0b4450 con 0x7f4c7c15e700 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c842d9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7c0d1d40 con 0x7f4c7c0d2360 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0b8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43888/0 (socket says 172.21.15.46:43888) 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 -- 172.21.15.46:0/3729077264 learned_addr learned my addr 172.21.15.46:0/3729077264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c81874640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c7c15e700 0x7f4c7c0b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82075640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c0d2360 0x7f4c7c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 -- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c0d2360 msgr2=0x7f4c7c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c0d2360 0x7f4c7c0ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 -- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c7c15e700 msgr2=0x7f4c7c0b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c7c15e700 0x7f4c7c0b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 -- 172.21.15.46:0/3729077264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c0ca950 con 0x7f4c7c15eae0 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c82876640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0b8e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009f40 tx=0x7f4c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74043020 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c7403e450 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c0cbdd0 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74009510 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.089+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c0cc390 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.090+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c0689e0 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.091+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c74046020 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.091+0000 7f4c72ffd640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c6006c8f0 0x7f4c6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.091+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4c740ae5a0 con 0x7f4c7c15eae0 2024-04-15T15:15:45.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.091+0000 7f4c82075640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c6006c8f0 0x7f4c6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.092+0000 7f4c82075640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c6006c8f0 0x7f4c6006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009930 tx=0x7f4c78008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.095+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c7404d030 con 0x7f4c7c15eae0 2024-04-15T15:15:45.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.280+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c7c0ba500 con 0x7f4c7c15eae0 2024-04-15T15:15:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.281+0000 7f4c72ffd640 1 -- 172.21.15.46:0/3729077264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f4c7407c040 con 0x7f4c7c15eae0 2024-04-15T15:15:45.283 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:45.283 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:45.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c6006c8f0 msgr2=0x7f4c6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c6006c8f0 0x7f4c6006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009930 tx=0x7f4c78008040 comp rx=0 tx=0).stop 2024-04-15T15:15:45.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 msgr2=0x7f4c7c0b8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0b8e60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009f40 tx=0x7f4c74004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:45.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 shutdown_connections 2024-04-15T15:15:45.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c6006c8f0 0x7f4c6006edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c7c15eae0 0x7f4c7c0b8e60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c7c15e700 0x7f4c7c0b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 --2- 172.21.15.46:0/3729077264 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c7c0d2360 0x7f4c7c0ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 >> 172.21.15.46:0/3729077264 conn(0x7f4c7c08bf60 msgr2=0x7f4c7c0cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 shutdown_connections 2024-04-15T15:15:45.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.283+0000 7f4c842d9640 1 -- 172.21.15.46:0/3729077264 wait complete. 2024-04-15T15:15:45.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:15:37.926112+0000', 'modified': '2024-04-15T15:15:42.956426+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4284}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4284': {'gid': 4284, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3854702652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3854702652}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3854702652}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4284, 'qdb_cluster': [4284]} 2024-04-15T15:15:45.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:15:45.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-15T15:15:45.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.805+0000 7f481a5f4640 1 Processor -- start 2024-04-15T15:15:45.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.805+0000 7f481a5f4640 1 -- start start 2024-04-15T15:15:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f481a5f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f481415e790 0x7f48140b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f481a5f4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481415eb70 con 0x7f48140b7be0 2024-04-15T15:15:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f481a5f4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48140d3c80 con 0x7f48140d1cf0 2024-04-15T15:15:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f481a5f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f48140d3790 con 0x7f481415e790 2024-04-15T15:15:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f48137fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f48140b7be0 0x7f48140d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49774/0 (socket says 172.21.15.46:49774) 2024-04-15T15:15:45.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f48137fe640 1 -- 172.21.15.46:0/2885058141 learned_addr learned my addr 172.21.15.46:0/2885058141 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4813fff640 1 --2- 172.21.15.46:0/2885058141 >> v2:172.21.15.78:3300/0 conn(0x7f481415e790 0x7f48140b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107012536 0 0) 0x7f481415eb70 con 0x7f48140b7be0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47fc003520 con 0x7f48140b7be0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961409047 0 0) 0x7f48140d3c80 con 0x7f48140d1cf0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f481415eb70 con 0x7f48140d1cf0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4105550735 0 0) 0x7f47fc003520 con 0x7f48140b7be0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 >> v2:172.21.15.78:3300/0 conn(0x7f481415e790 msgr2=0x7f48140b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 --2- 172.21.15.46:0/2885058141 >> v2:172.21.15.78:3300/0 conn(0x7f481415e790 0x7f48140b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 >> v1:172.21.15.78:6789/0 conn(0x7f48140d1cf0 legacy=0x7f48140d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48140d3f60 con 0x7f48140b7be0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f4808001690 con 0x7f48140b7be0 2024-04-15T15:15:45.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.806+0000 7f4813fff640 1 --2- 172.21.15.46:0/2885058141 >> v2:172.21.15.78:3300/0 conn(0x7f481415e790 0x7f48140b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 >> v1:172.21.15.46:6789/0 conn(0x7f48140b7be0 legacy=0x7f48140d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc003b40 0x7f47fc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f47fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f47fc005230 0x7f47fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48140d3790 con 0x7f47fc004800 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc00c190 con 0x7f47fc003b40 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc00c390 con 0x7f47fc005230 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f48137fe640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f47fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4813fff640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc003b40 0x7f47fc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4818b91640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f47fc005230 0x7f47fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f48137fe640 1 -- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f47fc005230 msgr2=0x7f47fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f48137fe640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f47fc005230 0x7f47fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f48137fe640 1 -- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc003b40 msgr2=0x7f47fc003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f48137fe640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f47fc003b40 0x7f47fc003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f48137fe640 1 -- 172.21.15.46:0/2885058141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc003520 con 0x7f47fc004800 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4818b91640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f47fc005230 0x7f47fc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.807+0000 7f4818b91640 1 -- 172.21.15.46:0/2885058141 reap_dead start 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f48137fe640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f47fc004c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f481415ed40 tx=0x7f47fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2d149d1494efa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808007ce0 con 0x7f47fc004800 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4808030070 con 0x7f47fc004800 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f4812ffd640 1 -- 172.21.15.46:0/2885058141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808005d30 con 0x7f47fc004800 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 -- 172.21.15.46:0/2885058141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 msgr2=0x7f47fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f47fc004c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f481415ed40 tx=0x7f47fc005830 comp rx=0 tx=0).stop 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 -- 172.21.15.46:0/2885058141 shutdown_connections 2024-04-15T15:15:45.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 --2- 172.21.15.46:0/2885058141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f47fc004c80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 -- 172.21.15.46:0/2885058141 >> 172.21.15.46:0/2885058141 conn(0x7f4814067b40 msgr2=0x7f4814067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 -- 172.21.15.46:0/2885058141 shutdown_connections 2024-04-15T15:15:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.808+0000 7f481a5f4640 1 -- 172.21.15.46:0/2885058141 wait complete. 2024-04-15T15:15:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 Processor -- start 2024-04-15T15:15:45.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 -- start start 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f48140c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48141f36b0 0x7f481415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f48141f3b10 0x7f481415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f48140b76a0 con 0x7f47fc004800 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f48140d45b0 con 0x7f48141f36b0 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.810+0000 7f481a5f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f48140b7820 con 0x7f48141f3b10 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f48140c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f48140c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43902/0 (socket says 172.21.15.46:43902) 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 -- 172.21.15.46:0/1771003829 learned_addr learned my addr 172.21.15.46:0/1771003829 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4818b91640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f48141f3b10 0x7f481415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f48137fe640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48141f36b0 0x7f481415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 -- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f48141f3b10 msgr2=0x7f481415ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f48141f3b10 0x7f481415ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 -- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48141f36b0 msgr2=0x7f481415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48141f36b0 0x7f481415a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 -- 172.21.15.46:0/1771003829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48140c8660 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4813fff640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f48140c8120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4804011d00 tx=0x7f480400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4810ff9640 1 -- 172.21.15.46:0/1771003829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480400fd20 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4810ff9640 1 -- 172.21.15.46:0/1771003829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4804004020 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4814160750 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f4810ff9640 1 -- 172.21.15.46:0/1771003829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804004410 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.811+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4814160d10 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.813+0000 7f4810ff9640 1 -- 172.21.15.46:0/1771003829 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4804012040 con 0x7f47fc004800 2024-04-15T15:15:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.814+0000 7f4810ff9640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f480006c8f0 0x7f480006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.814+0000 7f4810ff9640 1 -- 172.21.15.46:0/1771003829 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f480408c370 con 0x7f47fc004800 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.814+0000 7f48137fe640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f480006c8f0 0x7f480006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.814+0000 7f481d1e5740 1 Processor -- start 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.814+0000 7f481d1e5740 1 -- start start 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.814+0000 7f48137fe640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f480006c8f0 0x7f480006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4808034000 tx=0x7f4808039040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x56062627bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262abba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56062627c2f0 0x5606262ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f481d1e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5606261e69c0 con 0x5606262a91f0 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56062618d650 con 0x56062627c2f0 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x5606261e2980 con 0x5606262a8e10 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262abba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43910/0 (socket says 172.21.15.46:43910) 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 -- 172.21.15.46:0/1575252058 learned_addr learned my addr 172.21.15.46:0/1575252058 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f4818b91640 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56062627c2f0 0x5606262ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f4813fff640 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x56062627bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 -- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 msgr2=0x56062627bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x56062627bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 -- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56062627c2f0 msgr2=0x5606262ac0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56062627c2f0 0x5606262ac0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.815+0000 7f48137fe640 1 -- 172.21.15.46:0/1575252058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560625ef92d0 con 0x5606262a91f0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f48137fe640 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262abba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f480800c250 tx=0x7f480802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e2f58b2d44447 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f47f9ffb640 1 -- 172.21.15.46:0/1575252058 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48080048a0 con 0x5606262a91f0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f47f9ffb640 1 -- 172.21.15.46:0/1575252058 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4808030040 con 0x5606262a91f0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f47f9ffb640 1 -- 172.21.15.46:0/1575252058 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4808004c40 con 0x5606262a91f0 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 -- 172.21.15.46:0/1575252058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 msgr2=0x5606262abba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262abba0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f480800c250 tx=0x7f480802e000 comp rx=0 tx=0).stop 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 -- 172.21.15.46:0/1575252058 shutdown_connections 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56062627c2f0 0x5606262ac0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262abba0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 --2- 172.21.15.46:0/1575252058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x56062627bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.816+0000 7f481d1e5740 1 -- 172.21.15.46:0/1575252058 >> 172.21.15.46:0/1575252058 conn(0x56062627e020 msgr2=0x5606262c6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.817+0000 7f481d1e5740 1 -- 172.21.15.46:0/1575252058 shutdown_connections 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.817+0000 7f481d1e5740 1 -- 172.21.15.46:0/1575252058 wait complete. 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.817+0000 7f481d1e5740 1 -- start start 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.817+0000 7f481d1e5740 1 Processor -- start 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262a4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5606262805c0 con 0x5606262a4000 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5606261a0a10 con 0x5606262a8e10 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x56062627f9b0 con 0x5606262a91f0 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f481d1e5740 10 client.0 _ll_register_callbacks cb 0x5606261dc250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262a4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43916/0 (socket says 172.21.15.46:43916) 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 -- 172.21.15.46:0/3342908781 learned_addr learned my addr 172.21.15.46:0/3342908781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f48137fe640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4813fff640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 -- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 msgr2=0x5606262a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 -- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 msgr2=0x5606262a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.818+0000 7f4818b91640 1 -- 172.21.15.46:0/3342908781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560625fc36b0 con 0x5606262a4000 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f4818b91640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262a4480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f48140c80f0 tx=0x7f480c012980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19021e46d01b70d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f47f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f47f2ffd640 1 -- 172.21.15.46:0/3342908781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c019020 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f47f2ffd640 1 -- 172.21.15.46:0/3342908781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f480c003b80 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4808030040 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f47f2ffd640 1 -- 172.21.15.46:0/3342908781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f480c023050 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 10 client.4413 fetch_fsmap learned FSMap version 6 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x5606262da6f0 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f47f2ffd640 1 -- 172.21.15.46:0/3342908781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4808030040 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f47f2ffd640 1 -- 172.21.15.46:0/3342908781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f480c026020 con 0x5606262a4000 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 10 client.4413 fetch_fsmap finished waiting for FSMap version 6 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 10 client.4413 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 20 client.4413 populate_metadata read hostname 'smithi046' 2024-04-15T15:15:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d79a0 0x5606262e5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 4 client.4413 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 --> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606261f43e0 con 0x5606262d79a0 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.819+0000 7f4813fff640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d79a0 0x5606262e5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.820+0000 7f4813fff640 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d79a0 0x5606262e5b60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1655e189e73d2874 server_cookie=b7606ffb5b30da7d in_seq=0 out_seq=0 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.820+0000 7f47f2ffd640 10 client.4413 ms_handle_connect on v2:172.21.15.78:6835/3854702652 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f47f2ffd640 1 -- 172.21.15.46:0/3342908781 <== mds.0 v2:172.21.15.78:6835/3854702652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606261f43e0 con 0x5606262d79a0 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f47f2ffd640 10 client.4413 handle_command_reply: tid=0 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 client.4413 shutdown 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 msgr2=0x5606262a4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262a4480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f48140c80f0 tx=0x7f480c012980 comp rx=0 tx=0).stop 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 shutdown_connections 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d79a0 0x5606262e5b60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262a4480 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f47f2ffd640 0 client.4413 ms_handle_reset on v2:172.21.15.78:6835/3854702652 2024-04-15T15:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 --2- 172.21.15.46:0/3342908781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 >> 172.21.15.46:0/3342908781 conn(0x56062627c2f0 msgr2=0x5606262a7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 shutdown_connections 2024-04-15T15:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 1 -- 172.21.15.46:0/3342908781 wait complete. 2024-04-15T15:15:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.821+0000 7f481d1e5740 20 client.4413 trim_cache size 0 max 16384 2024-04-15T15:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.839+0000 7f481d1e5740 1 Processor -- start 2024-04-15T15:15:45.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.839+0000 7f481d1e5740 1 -- start start 2024-04-15T15:15:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a4000 0x5606262a2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x5606262eab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f481d1e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4814202930 con 0x5606262a91f0 2024-04-15T15:15:45.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4814201930 con 0x5606262a4000 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4814200c90 con 0x5606262a8e10 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262eb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43924/0 (socket says 172.21.15.46:43924) 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 -- 172.21.15.46:0/2629491134 learned_addr learned my addr 172.21.15.46:0/2629491134 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 -- 172.21.15.46:0/2629491134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 msgr2=0x5606262eab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x5606262eab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 -- 172.21.15.46:0/2629491134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a4000 msgr2=0x5606262a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a4000 0x5606262a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 -- 172.21.15.46:0/2629491134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606262ad8e0 con 0x5606262a91f0 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f4818b91640 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262eb0d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f480c001540 tx=0x7f480c00d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6497585085f26832 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f47f97fa640 1 -- 172.21.15.46:0/2629491134 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c02b070 con 0x5606262a91f0 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f47f97fa640 1 -- 172.21.15.46:0/2629491134 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c003e50 con 0x5606262a91f0 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.840+0000 7f47f97fa640 1 -- 172.21.15.46:0/2629491134 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c019020 con 0x5606262a91f0 2024-04-15T15:15:45.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 -- 172.21.15.46:0/2629491134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 msgr2=0x5606262eb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262eb0d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f480c001540 tx=0x7f480c00d000 comp rx=0 tx=0).stop 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 -- 172.21.15.46:0/2629491134 shutdown_connections 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a91f0 0x5606262eb0d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a8e10 0x5606262eab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2629491134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a4000 0x5606262a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 -- 172.21.15.46:0/2629491134 >> 172.21.15.46:0/2629491134 conn(0x56062627c2f0 msgr2=0x560626282dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 -- 172.21.15.46:0/2629491134 shutdown_connections 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 -- 172.21.15.46:0/2629491134 wait complete. 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.841+0000 7f481d1e5740 1 -- start start 2024-04-15T15:15:45.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 Processor -- start 2024-04-15T15:15:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262ea670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5606262dae40 con 0x5606262a4000 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f48142012e0 con 0x5606262a8e10 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.842+0000 7f481d1e5740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x5606262a49c0 con 0x5606262a91f0 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f481d1e5740 10 client.0 _ll_register_callbacks cb 0x7f480c007db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262ea670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262ea670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43932/0 (socket says 172.21.15.46:43932) 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 -- 172.21.15.46:0/2680569250 learned_addr learned my addr 172.21.15.46:0/2680569250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4818b91640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 -- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 msgr2=0x5606262a6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f48137fe640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 -- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 msgr2=0x5606262c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 -- 172.21.15.46:0/2680569250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606262eabb0 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f4813fff640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262ea670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4804004650 tx=0x7f480408c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8531d0d37e44ebce server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f47f2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f47f2ffd640 1 -- 172.21.15.46:0/2680569250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4804022040 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f47f2ffd640 1 -- 172.21.15.46:0/2680569250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f480408c0b0 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f480c003e50 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f47f2ffd640 1 -- 172.21.15.46:0/2680569250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f480405e110 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f481d1e5740 10 client.4419 fetch_fsmap learned FSMap version 6 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x5606262a7930 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f47f2ffd640 1 -- 172.21.15.46:0/2680569250 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c003e50 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f47f2ffd640 1 -- 172.21.15.46:0/2680569250 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f48040563d0 con 0x5606262a4000 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f481d1e5740 10 client.4419 fetch_fsmap finished waiting for FSMap version 6 2024-04-15T15:15:45.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.843+0000 7f481d1e5740 10 client.4419 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 20 client.4419 populate_metadata read hostname 'smithi046' 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d5ad0 0x560626344f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 4 client.4419 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 --> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] -- command(tid 0: {"prefix": "status"}) -- 0x5606261ee250 con 0x5606262d5ad0 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f48137fe640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d5ad0 0x560626344f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f47f2ffd640 10 client.4419 ms_handle_connect on v2:172.21.15.78:6835/3854702652 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f48137fe640 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d5ad0 0x560626344f30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aabcec4707875dd9 server_cookie=e632910d6adb94aa in_seq=0 out_seq=0 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f47f2ffd640 1 -- 172.21.15.46:0/2680569250 <== mds.0 v2:172.21.15.78:6835/3854702652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5606261ee250 con 0x5606262d5ad0 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f47f2ffd640 10 client.4419 handle_command_reply: tid=0 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 client.4419 shutdown 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 msgr2=0x5606262ea670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262ea670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4804004650 tx=0x7f480408c040 comp rx=0 tx=0).stop 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 shutdown_connections 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x5606262d5ad0 0x560626344f30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5606262a91f0 0x5606262a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5606262a8e10 0x5606262c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f47f2ffd640 0 client.4419 ms_handle_reset on v2:172.21.15.78:6835/3854702652 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 --2- 172.21.15.46:0/2680569250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5606262a4000 0x5606262ea670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.844+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 >> 172.21.15.46:0/2680569250 conn(0x56062627c2f0 msgr2=0x5606262a0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 shutdown_connections 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481d1e5740 1 -- 172.21.15.46:0/2680569250 wait complete. 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481d1e5740 20 client.4419 trim_cache size 0 max 16384 2024-04-15T15:15:45.853 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "95f1e89e-6559-430a-8622-bd018dcc0afd", 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4284, 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 7.2907329760000001, 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 6, 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 25, 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 24, 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 13.868754559999999 2024-04-15T15:15:45.854 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:15:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f480006c8f0 msgr2=0x7f480006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f480006c8f0 0x7f480006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4808034000 tx=0x7f4808039040 comp rx=0 tx=0).stop 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 msgr2=0x7f48140c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f48140c8120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4804011d00 tx=0x7f480400f610 comp rx=0 tx=0).stop 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 shutdown_connections 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f480006c8f0 0x7f480006edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f48141f3b10 0x7f481415ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48141f36b0 0x7f481415a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 --2- 172.21.15.46:0/1771003829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47fc004800 0x7f48140c8120 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 >> 172.21.15.46:0/1771003829 conn(0x7f4814067b40 msgr2=0x7f48140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 shutdown_connections 2024-04-15T15:15:45.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.845+0000 7f481a5f4640 1 -- 172.21.15.46:0/1771003829 wait complete. 2024-04-15T15:15:45.857 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:45.858 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:45.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.961+0000 7fe2ece88640 1 Processor -- start 2024-04-15T15:15:45.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.961+0000 7fe2ece88640 1 -- start start 2024-04-15T15:15:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2ece88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e80b4b40 0x7fe2e80b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2ece88640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2e80b0800 0x7fe2e80b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2ece88640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2e80af220 con 0x7fe2e815b4d0 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2ece88640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe2e80af500 con 0x7fe2e80b0800 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2ece88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e80af7e0 con 0x7fe2e80b4b40 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2e6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2e80b0800 0x7fe2e80b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2e659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2e815b4d0 0x7fe2e80ae960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49780/0 (socket says 172.21.15.46:49780) 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.962+0000 7fe2e659c640 1 -- 172.21.15.46:0/2683115630 learned_addr learned my addr 172.21.15.46:0/2683115630 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e5d9b640 1 --2- 172.21.15.46:0/2683115630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e80b4b40 0x7fe2e80b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/2683115630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e80b4b40 msgr2=0x7fe2e80b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e6d9d640 1 --2- 172.21.15.46:0/2683115630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e80b4b40 0x7fe2e80b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/2683115630 >> v1:172.21.15.46:6789/0 conn(0x7fe2e815b4d0 legacy=0x7fe2e80ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/2683115630 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e80afac0 con 0x7fe2e80b0800 2024-04-15T15:15:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e559a640 1 -- 172.21.15.46:0/2683115630 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324638617 0 0) 0x7fe2e80af220 con 0x7fe2e815b4d0 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.963+0000 7fe2e6d9d640 1 --2- 172.21.15.46:0/2683115630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80b0800 0x7fe2e80b21e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197c25e3e546fbb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2e559a640 1 -- 172.21.15.46:0/2683115630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8039070 con 0x7fe2e80b0800 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2e559a640 1 -- 172.21.15.46:0/2683115630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d802cc50 con 0x7fe2e80b0800 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2e559a640 1 -- 172.21.15.46:0/2683115630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8034440 con 0x7fe2e80b0800 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 -- 172.21.15.46:0/2683115630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80b0800 msgr2=0x7fe2e80b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 --2- 172.21.15.46:0/2683115630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80b0800 0x7fe2e80b21e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d802bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 -- 172.21.15.46:0/2683115630 shutdown_connections 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 --2- 172.21.15.46:0/2683115630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80b0800 0x7fe2e80b21e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 --2- 172.21.15.46:0/2683115630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e80b4b40 0x7fe2e80b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 -- 172.21.15.46:0/2683115630 >> 172.21.15.46:0/2683115630 conn(0x7fe2e8073260 msgr2=0x7fe2e809a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 -- 172.21.15.46:0/2683115630 shutdown_connections 2024-04-15T15:15:45.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.964+0000 7fe2ece88640 1 -- 172.21.15.46:0/2683115630 wait complete. 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.965+0000 7fe2ece88640 1 Processor -- start 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 -- start start 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e80b4b40 0x7fe2e814f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e815b4d0 0x7fe2e80cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 0x7fe2e806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e80b28a0 con 0x7fe2e80cd760 2024-04-15T15:15:45.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e80b2720 con 0x7fe2e80b4b40 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2ece88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe2e80b2a20 con 0x7fe2e815b4d0 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2e6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 0x7fe2e806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2e6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 0x7fe2e806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43960/0 (socket says 172.21.15.46:43960) 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/1773528564 learned_addr learned my addr 172.21.15.46:0/1773528564 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.966+0000 7fe2e5d9b640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e815b4d0 0x7fe2e80cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e815b4d0 msgr2=0x7fe2e80cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2e6d9d640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e815b4d0 0x7fe2e80cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e80b4b40 msgr2=0x7fe2e814f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2e6d9d640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e80b4b40 0x7fe2e814f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2e6d9d640 1 -- 172.21.15.46:0/1773528564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e8070720 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2e6d9d640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 0x7fe2e806b9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8039050 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d8045070 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e8070a00 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d803e610 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.967+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e809b640 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.968+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e80690f0 con 0x7fe2e80cd760 2024-04-15T15:15:45.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.968+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2d803e7b0 con 0x7fe2e80cd760 2024-04-15T15:15:45.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.969+0000 7fe2deffd640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2c806c920 0x7fe2c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.969+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe2d80ae9b0 con 0x7fe2e80cd760 2024-04-15T15:15:45.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.969+0000 7fe2e659c640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2c806c920 0x7fe2c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.970+0000 7fe2e659c640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2c806c920 0x7fe2c806ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0007fd0 tx=0x7fe2e0007960 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:45.973+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2d807ca90 con 0x7fe2e80cd760 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.198+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe2e8070bd0 con 0x7fe2e80cd760 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.199+0000 7fe2deffd640 1 -- 172.21.15.46:0/1773528564 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7fe2d807c390 con 0x7fe2e80cd760 2024-04-15T15:15:46.201 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2c806c920 msgr2=0x7fe2c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2c806c920 0x7fe2c806ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0007fd0 tx=0x7fe2e0007960 comp rx=0 tx=0).stop 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 msgr2=0x7fe2e806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 0x7fe2e806b9e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009d90 tx=0x7fe2d8004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 shutdown_connections 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2c806c920 0x7fe2c806ede0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2e80cd760 0x7fe2e806b9e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2e815b4d0 0x7fe2e80cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 --2- 172.21.15.46:0/1773528564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2e80b4b40 0x7fe2e814f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.200+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 >> 172.21.15.46:0/1773528564 conn(0x7fe2e8073260 msgr2=0x7fe2e80b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.201+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 shutdown_connections 2024-04-15T15:15:46.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46.201+0000 7fe2ece88640 1 -- 172.21.15.46:0/1773528564 wait complete. 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-15T15:15:46.213 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:46.214 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:46.214 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:15:46.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:46.214 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-15T15:15:46.230 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:46.230 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:46.230 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:46.230 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-15T15:15:46.231 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:15:46.232 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42706sec preferred_lft 42706sec 2024-04-15T15:15:46.232 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-15T15:15:46.232 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:46.232 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:46.232 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:46.232 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-15T15:15:46.233 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:15:46.233 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:46.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:46.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:46.437 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:46.466 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-15T15:15:46.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:46.468 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-15T15:15:46.539 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-15T15:15:46.539 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:46.540 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:15:46.540 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:15:46.540 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:46.540 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:46.540 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:15:46.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:15:46.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:15:46.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:15:46.541 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:15:46.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:47.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:47.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:47.249 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:15:47.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:47.268 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-04-15T15:15:47.327 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:47.327 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:15:47.327 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:47.327 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-15T15:15:47.327 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:15:47.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:47.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:47.460 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-15T15:15:47.460 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:15:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:47.606 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:47.606 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:15:47.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-04-15T15:15:47.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:47.606 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:15:47.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:47.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:47.690 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-15T15:15:47.690 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:47.690 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:47.705 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:15:47.705 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:47.706 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:47.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:15:47.829 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:15:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:15:47.889 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:47.890 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:47.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:47.891 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:15:47.941 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:47.942 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-15T15:15:47.984 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:15:48.012 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:48.068 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:15:48.069 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:48.070 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:48.071 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:48.071 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:48.071 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:15:48.071 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:48.071 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:48.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:48.071 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:15:48.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:2024-04-15T15:15:48.073+0000 7fbfbac6d480 -1 init, newargv = 0x55de575eeec0 newargc=15 2024-04-15T15:15:48.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[44463]: starting ceph client 2024-04-15T15:15:48.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[44463]: starting fuse 2024-04-15T15:15:48.090 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:15:48.090 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:15:48.093 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:48.093 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:15:48.094 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.admin")) 2024-04-15T15:15:48.095 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-04-15T15:15:48.213 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.admin.44463.asok 2024-04-15T15:15:48.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:48.224 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44463.asok 2024-04-15T15:15:48.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:48.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44463.asok status 2024-04-15T15:15:48.363 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:15:48.363 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-04-15T15:15:48.363 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:48.363 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "admin", 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "44463", 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4437, 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-04-15T15:15:48.364 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4437 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1776731640 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:15:48.365 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1776731640 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4437 192.168.0.1:0/1776731640", 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1776731640", 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 6, 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 25, 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-04-15T15:15:48.366 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:15:48.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:48.372 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-04-15T15:15:48.372 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:48.387 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-04-15T15:15:48.388 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:48.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:48.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:48.457 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-15T15:15:48.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:15:48.576 INFO:teuthology.orchestra.run.smithi046.stdout:# file: . 2024-04-15T15:15:48.576 INFO:teuthology.orchestra.run.smithi046.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-15T15:15:48.576 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:48.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:48.582 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:48.582 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:48.598 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.740+0000 7fe1370d3640 1 Processor -- start 2024-04-15T15:15:48.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.740+0000 7fe1370d3640 1 -- start start 2024-04-15T15:15:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe1370d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1300d0da0 0x7fe13006e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe1370d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe13015db40 con 0x7fe130072d80 2024-04-15T15:15:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe1370d3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe130072170 con 0x7fe130071d10 2024-04-15T15:15:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe1370d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe13015cfc0 con 0x7fe1300d0da0 2024-04-15T15:15:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe135670640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1300d0da0 0x7fe13006e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe134e6f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe130072d80 0x7fe1300af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49784/0 (socket says 172.21.15.46:49784) 2024-04-15T15:15:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe135670640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe1300d0da0 0x7fe13006e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43972/0 (socket says 172.21.15.46:43972) 2024-04-15T15:15:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe134e6f640 1 -- 172.21.15.46:0/3902945048 learned_addr learned my addr 172.21.15.46:0/3902945048 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe135670640 1 -- 172.21.15.46:0/3902945048 >> v1:172.21.15.78:6789/0 conn(0x7fe130071d10 legacy=0x7fe1300d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe135670640 1 -- 172.21.15.46:0/3902945048 >> v1:172.21.15.46:6789/0 conn(0x7fe130072d80 legacy=0x7fe1300af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.741+0000 7fe135670640 1 -- 172.21.15.46:0/3902945048 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13015d2a0 con 0x7fe1300d0da0 2024-04-15T15:15:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.742+0000 7fe135670640 1 --2- 172.21.15.46:0/3902945048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300d0da0 0x7fe13006e060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe130072340 tx=0x7fe12802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bc8940f8bd4d4e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.742+0000 7fe1277fe640 1 -- 172.21.15.46:0/3902945048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe128039070 con 0x7fe1300d0da0 2024-04-15T15:15:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.742+0000 7fe1277fe640 1 -- 172.21.15.46:0/3902945048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12802cbc0 con 0x7fe1300d0da0 2024-04-15T15:15:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.742+0000 7fe1277fe640 1 -- 172.21.15.46:0/3902945048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe128034440 con 0x7fe1300d0da0 2024-04-15T15:15:48.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 -- 172.21.15.46:0/3902945048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300d0da0 msgr2=0x7fe13006e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 --2- 172.21.15.46:0/3902945048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300d0da0 0x7fe13006e060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe130072340 tx=0x7fe12802be10 comp rx=0 tx=0).stop 2024-04-15T15:15:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 -- 172.21.15.46:0/3902945048 shutdown_connections 2024-04-15T15:15:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 --2- 172.21.15.46:0/3902945048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300d0da0 0x7fe13006e060 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 -- 172.21.15.46:0/3902945048 >> 172.21.15.46:0/3902945048 conn(0x7fe13006be00 msgr2=0x7fe13006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 -- 172.21.15.46:0/3902945048 shutdown_connections 2024-04-15T15:15:48.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.743+0000 7fe1370d3640 1 -- 172.21.15.46:0/3902945048 wait complete. 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.744+0000 7fe1370d3640 1 Processor -- start 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.744+0000 7fe1370d3640 1 -- start start 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe1370d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 0x7fe13014c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe1370d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130072d80 0x7fe1300ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe1370d3640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1300d0da0 0x7fe13014d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe1370d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe1300afaf0 con 0x7fe130071d10 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe1370d3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe13015d860 con 0x7fe130072d80 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe1370d3640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe13015cb90 con 0x7fe1300d0da0 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 0x7fe13014c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 0x7fe13014c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43984/0 (socket says 172.21.15.46:43984) 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 -- 172.21.15.46:0/569288330 learned_addr learned my addr 172.21.15.46:0/569288330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe127fff640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130072d80 0x7fe1300ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe135670640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1300d0da0 0x7fe13014d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 -- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1300d0da0 msgr2=0x7fe13014d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1300d0da0 0x7fe13014d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 -- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130072d80 msgr2=0x7fe1300ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130072d80 0x7fe1300ce5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 -- 172.21.15.46:0/569288330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1301573f0 con 0x7fe130071d10 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe135670640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1300d0da0 0x7fe13014d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.745+0000 7fe134e6f640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 0x7fe13014c310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00c400 tx=0x7fe12c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.746+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c017070 con 0x7fe130071d10 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.746+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe12c005ce0 con 0x7fe130071d10 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.746+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1301576d0 con 0x7fe130071d10 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.746+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c0072f0 con 0x7fe130071d10 2024-04-15T15:15:48.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.746+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe130157c90 con 0x7fe130071d10 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.747+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1300689e0 con 0x7fe130071d10 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.747+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe12c01a070 con 0x7fe130071d10 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.748+0000 7fe1257fa640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.748+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe12c08b2a0 con 0x7fe130071d10 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.748+0000 7fe127fff640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.748+0000 7fe127fff640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe11c06c8f0 0x7fe11c06edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe1300cc350 tx=0x7fe120009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.752+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe12c059390 con 0x7fe130071d10 2024-04-15T15:15:48.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.936+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1300ad360 con 0x7fe130071d10 2024-04-15T15:15:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.937+0000 7fe1257fa640 1 -- 172.21.15.46:0/569288330 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fe12c058c90 con 0x7fe130071d10 2024-04-15T15:15:48.939 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:48.940 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe11c06c8f0 msgr2=0x7fe11c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe11c06c8f0 0x7fe11c06edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe1300cc350 tx=0x7fe120009040 comp rx=0 tx=0).stop 2024-04-15T15:15:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 msgr2=0x7fe13014c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 0x7fe13014c310 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00c400 tx=0x7fe12c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 shutdown_connections 2024-04-15T15:15:48.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe11c06c8f0 0x7fe11c06edb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1300d0da0 0x7fe13014d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130072d80 0x7fe1300ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 --2- 172.21.15.46:0/569288330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe130071d10 0x7fe13014c310 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 >> 172.21.15.46:0/569288330 conn(0x7fe13006be00 msgr2=0x7fe130074de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 shutdown_connections 2024-04-15T15:15:48.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:48.939+0000 7fe1370d3640 1 -- 172.21.15.46:0/569288330 wait complete. 2024-04-15T15:15:48.952 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:48.953 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:49.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.058+0000 7fdd5b300640 1 Processor -- start 2024-04-15T15:15:49.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.058+0000 7fdd5b300640 1 -- start start 2024-04-15T15:15:49.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5b300640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd5415e700 0x7fdd540b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5b300640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdd540d2360 0x7fdd540d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5b300640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415eae0 0x7fdd540d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5b300640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdd540b53b0 con 0x7fdd5415e700 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5b300640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fdd540b47a0 con 0x7fdd540d2360 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5b300640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdd540b4a80 con 0x7fdd5415eae0 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5909c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd5415e700 0x7fdd540b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5909c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd5415e700 0x7fdd540b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44000/0 (socket says 172.21.15.46:44000) 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.059+0000 7fdd5909c640 1 -- 172.21.15.46:0/1289981781 learned_addr learned my addr 172.21.15.46:0/1289981781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5989d640 1 --2- 172.21.15.46:0/1289981781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415eae0 0x7fdd540d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5889b640 1 --2- 172.21.15.46:0/1289981781 >> v2:172.21.15.78:3300/0 conn(0x7fdd540d2360 0x7fdd540d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5909c640 1 -- 172.21.15.46:0/1289981781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415eae0 msgr2=0x7fdd540d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5909c640 1 --2- 172.21.15.46:0/1289981781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415eae0 0x7fdd540d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5909c640 1 -- 172.21.15.46:0/1289981781 >> v2:172.21.15.78:3300/0 conn(0x7fdd540d2360 msgr2=0x7fdd540d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5909c640 1 --2- 172.21.15.46:0/1289981781 >> v2:172.21.15.78:3300/0 conn(0x7fdd540d2360 0x7fdd540d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5909c640 1 -- 172.21.15.46:0/1289981781 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd540b4d60 con 0x7fdd5415e700 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5889b640 1 --2- 172.21.15.46:0/1289981781 >> v2:172.21.15.78:3300/0 conn(0x7fdd540d2360 0x7fdd540d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.060+0000 7fdd5909c640 1 --2- 172.21.15.46:0/1289981781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd5415e700 0x7fdd540b5840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd5408c7b0 tx=0x7fdd4402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515b5c2ec4960ca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd53fff640 1 -- 172.21.15.46:0/1289981781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4402cac0 con 0x7fdd5415e700 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd53fff640 1 -- 172.21.15.46:0/1289981781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4402cc60 con 0x7fdd5415e700 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd53fff640 1 -- 172.21.15.46:0/1289981781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44034610 con 0x7fdd5415e700 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 -- 172.21.15.46:0/1289981781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd5415e700 msgr2=0x7fdd540b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 --2- 172.21.15.46:0/1289981781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd5415e700 0x7fdd540b5840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd5408c7b0 tx=0x7fdd4402bf20 comp rx=0 tx=0).stop 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 -- 172.21.15.46:0/1289981781 shutdown_connections 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 --2- 172.21.15.46:0/1289981781 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415eae0 0x7fdd540d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 --2- 172.21.15.46:0/1289981781 >> v2:172.21.15.78:3300/0 conn(0x7fdd540d2360 0x7fdd540d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 --2- 172.21.15.46:0/1289981781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd5415e700 0x7fdd540b5840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 -- 172.21.15.46:0/1289981781 >> 172.21.15.46:0/1289981781 conn(0x7fdd5408bf60 msgr2=0x7fdd540d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 -- 172.21.15.46:0/1289981781 shutdown_connections 2024-04-15T15:15:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.061+0000 7fdd5b300640 1 -- 172.21.15.46:0/1289981781 wait complete. 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.062+0000 7fdd5b300640 1 Processor -- start 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 -- start start 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 0x7fdd540c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415e700 0x7fdd541e8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5415eae0 0x7fdd541e8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd540b5d80 con 0x7fdd540d2360 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdd540b4450 con 0x7fdd5415eae0 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5b300640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdd540d1d40 con 0x7fdd5415e700 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5909c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 0x7fdd540c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5909c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 0x7fdd540c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44008/0 (socket says 172.21.15.46:44008) 2024-04-15T15:15:49.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.063+0000 7fdd5909c640 1 -- 172.21.15.46:0/3607242718 learned_addr learned my addr 172.21.15.46:0/3607242718 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5889b640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415e700 0x7fdd541e8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5989d640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5415eae0 0x7fdd541e8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5909c640 1 -- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415e700 msgr2=0x7fdd541e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5909c640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415e700 0x7fdd541e8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5909c640 1 -- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5415eae0 msgr2=0x7fdd541e8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5909c640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5415eae0 0x7fdd541e8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5909c640 1 -- 172.21.15.46:0/3607242718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd540b9260 con 0x7fdd540d2360 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5909c640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 0x7fdd540c7060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdd44036040 tx=0x7fdd44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44043020 con 0x7fdd540d2360 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4403e450 con 0x7fdd540d2360 2024-04-15T15:15:49.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd540b9540 con 0x7fdd540d2360 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd44009510 con 0x7fdd540d2360 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.064+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd540b9b00 con 0x7fdd540d2360 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.065+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd540689e0 con 0x7fdd540d2360 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.066+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd44046020 con 0x7fdd540d2360 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.066+0000 7fdd51ffb640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdd4006c920 0x7fdd4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.066+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdd440ae730 con 0x7fdd540d2360 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.067+0000 7fdd5889b640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdd4006c920 0x7fdd4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.067+0000 7fdd5889b640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdd4006c920 0x7fdd4006ede0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdd5406c520 tx=0x7fdd48008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.071+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd4404d030 con 0x7fdd540d2360 2024-04-15T15:15:49.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.253+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd540ce9b0 con 0x7fdd540d2360 2024-04-15T15:15:49.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.253+0000 7fdd51ffb640 1 -- 172.21.15.46:0/3607242718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fdd4407c820 con 0x7fdd540d2360 2024-04-15T15:15:49.256 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdd4006c920 msgr2=0x7fdd4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdd4006c920 0x7fdd4006ede0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdd5406c520 tx=0x7fdd48008040 comp rx=0 tx=0).stop 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 msgr2=0x7fdd540c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 0x7fdd540c7060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdd44036040 tx=0x7fdd44004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 shutdown_connections 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdd4006c920 0x7fdd4006ede0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdd5415eae0 0x7fdd541e8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdd5415e700 0x7fdd541e8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 --2- 172.21.15.46:0/3607242718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd540d2360 0x7fdd540c7060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 >> 172.21.15.46:0/3607242718 conn(0x7fdd5408bf60 msgr2=0x7fdd5415fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 shutdown_connections 2024-04-15T15:15:49.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.256+0000 7fdd5b300640 1 -- 172.21.15.46:0/3607242718 wait complete. 2024-04-15T15:15:49.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:49.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.374+0000 7f9a8a888640 1 Processor -- start 2024-04-15T15:15:49.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.374+0000 7f9a8a888640 1 -- start start 2024-04-15T15:15:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a8a888640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a8415b690 0x7f9a840b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a8a888640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8415ba70 con 0x7f9a840d08a0 2024-04-15T15:15:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a8a888640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a840ce5b0 con 0x7f9a840d1a10 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a8a888640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9a840ce890 con 0x7f9a8415b690 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a837fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9a840d08a0 0x7f9a840d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49792/0 (socket says 172.21.15.46:49792) 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a837fe640 1 -- 172.21.15.46:0/3601742058 learned_addr learned my addr 172.21.15.46:0/3601742058 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.375+0000 7f9a83fff640 1 --2- 172.21.15.46:0/3601742058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a8415b690 0x7f9a840b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2503997097 0 0) 0x7f9a78001110 con 0x7f9a840d08a0 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a6c003520 con 0x7f9a840d08a0 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690144846 0 0) 0x7f9a840ce5b0 con 0x7f9a840d1a10 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a78001110 con 0x7f9a840d1a10 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a83fff640 1 -- 172.21.15.46:0/3601742058 >> v1:172.21.15.78:6789/0 conn(0x7f9a840d1a10 legacy=0x7f9a840b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a83fff640 1 -- 172.21.15.46:0/3601742058 >> v1:172.21.15.46:6789/0 conn(0x7f9a840d08a0 legacy=0x7f9a840d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.376+0000 7f9a83fff640 1 -- 172.21.15.46:0/3601742058 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a840ceb70 con 0x7f9a8415b690 2024-04-15T15:15:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a83fff640 1 --2- 172.21.15.46:0/3601742058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a8415b690 0x7f9a840b6ff0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097d0 tx=0x7f9a7402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73717622ee41cdc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74035020 con 0x7f9a8415b690 2024-04-15T15:15:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a74031890 con 0x7f9a8415b690 2024-04-15T15:15:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a82ffd640 1 -- 172.21.15.46:0/3601742058 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74031c40 con 0x7f9a8415b690 2024-04-15T15:15:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 -- 172.21.15.46:0/3601742058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a8415b690 msgr2=0x7f9a840b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 --2- 172.21.15.46:0/3601742058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a8415b690 0x7f9a840b6ff0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097d0 tx=0x7f9a7402b930 comp rx=0 tx=0).stop 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 -- 172.21.15.46:0/3601742058 shutdown_connections 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 --2- 172.21.15.46:0/3601742058 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a8415b690 0x7f9a840b6ff0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 -- 172.21.15.46:0/3601742058 >> 172.21.15.46:0/3601742058 conn(0x7f9a8409a9d0 msgr2=0x7f9a840b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 -- 172.21.15.46:0/3601742058 shutdown_connections 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.377+0000 7f9a8a888640 1 -- 172.21.15.46:0/3601742058 wait complete. 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.379+0000 7f9a8a888640 1 Processor -- start 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.379+0000 7f9a8a888640 1 -- start start 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.379+0000 7f9a8a888640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 0x7f9a841e3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.379+0000 7f9a8a888640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a840d1a10 0x7f9a841532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.379+0000 7f9a8a888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a8415b690 0x7f9a84153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a8a888640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9a840b1d70 con 0x7f9a840d08a0 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a8a888640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9a840d1010 con 0x7f9a8415b690 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a8a888640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9a840b1ef0 con 0x7f9a840d1a10 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 0x7f9a841e3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 0x7f9a841e3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44010/0 (socket says 172.21.15.46:44010) 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 -- 172.21.15.46:0/1904460464 learned_addr learned my addr 172.21.15.46:0/1904460464 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a837fe640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a840d1a10 0x7f9a841532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a88e25640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a8415b690 0x7f9a84153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 -- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a840d1a10 msgr2=0x7f9a841532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a840d1a10 0x7f9a841532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 -- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a8415b690 msgr2=0x7f9a84153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a8415b690 0x7f9a84153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 -- 172.21.15.46:0/1904460464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a84153f30 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a88e25640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a8415b690 0x7f9a84153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a83fff640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 0x7f9a841e3220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9a74034040 tx=0x7f9a740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74003820 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a740039c0 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a84154210 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74003d70 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.380+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a841547d0 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.381+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a840690f0 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.382+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a7404c060 con 0x7f9a840d08a0 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.382+0000 7f9a80ff9640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9a7006c8f0 0x7f9a7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.382+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9a740aded0 con 0x7f9a840d08a0 2024-04-15T15:15:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.382+0000 7f9a837fe640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9a7006c8f0 0x7f9a7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.383+0000 7f9a837fe640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9a7006c8f0 0x7f9a7006edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9a841f5840 tx=0x7f9a78007d70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.387+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a74040070 con 0x7f9a840d08a0 2024-04-15T15:15:49.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.571+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a84154bf0 con 0x7f9a840d08a0 2024-04-15T15:15:49.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.572+0000 7f9a80ff9640 1 -- 172.21.15.46:0/1904460464 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f9a7402d040 con 0x7f9a840d08a0 2024-04-15T15:15:49.574 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9a7006c8f0 msgr2=0x7f9a7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9a7006c8f0 0x7f9a7006edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9a841f5840 tx=0x7f9a78007d70 comp rx=0 tx=0).stop 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 msgr2=0x7f9a841e3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 0x7f9a841e3220 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9a74034040 tx=0x7f9a740046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 shutdown_connections 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9a7006c8f0 0x7f9a7006edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a8415b690 0x7f9a84153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a840d1a10 0x7f9a841532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 --2- 172.21.15.46:0/1904460464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a840d08a0 0x7f9a841e3220 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 >> 172.21.15.46:0/1904460464 conn(0x7f9a8409a9d0 msgr2=0x7f9a841518a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 shutdown_connections 2024-04-15T15:15:49.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.574+0000 7f9a8a888640 1 -- 172.21.15.46:0/1904460464 wait complete. 2024-04-15T15:15:49.587 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.693+0000 7fde7409a640 1 Processor -- start 2024-04-15T15:15:49.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.693+0000 7fde7409a640 1 -- start start 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde7409a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde6c15e790 0x7fde6c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde7409a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde6c15eb70 con 0x7fde6c0b7be0 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde7409a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde6c0d3c80 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde7409a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fde6c0d3790 con 0x7fde6c15e790 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde71e36640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fde6c15e790 0x7fde6c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde71635640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fde6c0b7be0 0x7fde6c0d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41498/0 (socket says 172.21.15.46:41498) 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde71635640 1 -- 172.21.15.46:0/3031635102 learned_addr learned my addr 172.21.15.46:0/3031635102 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde71e36640 1 -- 172.21.15.46:0/3031635102 >> v1:172.21.15.78:6789/0 conn(0x7fde6c0d1cf0 legacy=0x7fde6c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde71e36640 1 -- 172.21.15.46:0/3031635102 >> v1:172.21.15.46:6789/0 conn(0x7fde6c0b7be0 legacy=0x7fde6c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde70e34640 1 -- 172.21.15.46:0/3031635102 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3327272787 0 0) 0x7fde6c15eb70 con 0x7fde6c0b7be0 2024-04-15T15:15:49.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.694+0000 7fde71e36640 1 -- 172.21.15.46:0/3031635102 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c0d3f60 con 0x7fde6c15e790 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.695+0000 7fde71e36640 1 --2- 172.21.15.46:0/3031635102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c15e790 0x7fde6c0b76a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde64009d90 tx=0x7fde6402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc358155c8a9331b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.695+0000 7fde70e34640 1 -- 172.21.15.46:0/3031635102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64039070 con 0x7fde6c15e790 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.695+0000 7fde70e34640 1 -- 172.21.15.46:0/3031635102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6402cb40 con 0x7fde6c15e790 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.695+0000 7fde70e34640 1 -- 172.21.15.46:0/3031635102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde64034440 con 0x7fde6c15e790 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 -- 172.21.15.46:0/3031635102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c15e790 msgr2=0x7fde6c0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 --2- 172.21.15.46:0/3031635102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c15e790 0x7fde6c0b76a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde64009d90 tx=0x7fde6402bd90 comp rx=0 tx=0).stop 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 -- 172.21.15.46:0/3031635102 shutdown_connections 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 --2- 172.21.15.46:0/3031635102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c15e790 0x7fde6c0b76a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 -- 172.21.15.46:0/3031635102 >> 172.21.15.46:0/3031635102 conn(0x7fde6c067b40 msgr2=0x7fde6c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 -- 172.21.15.46:0/3031635102 shutdown_connections 2024-04-15T15:15:49.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.696+0000 7fde7409a640 1 -- 172.21.15.46:0/3031635102 wait complete. 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.697+0000 7fde7409a640 1 Processor -- start 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.697+0000 7fde7409a640 1 -- start start 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde7409a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fde6c0b7be0 0x7fde6c1548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde7409a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 0x7fde6c15d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde7409a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6c15e790 0x7fde6c15d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde7409a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c0d5150 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde7409a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c0d45b0 con 0x7fde6c15e790 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde7409a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fde6c0d52d0 con 0x7fde6c0b7be0 2024-04-15T15:15:49.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 0x7fde6c15d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 0x7fde6c15d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60246/0 (socket says 172.21.15.46:60246) 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 -- 172.21.15.46:0/588110401 learned_addr learned my addr 172.21.15.46:0/588110401 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71e36640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fde6c0b7be0 0x7fde6c1548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde72637640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6c15e790 0x7fde6c15d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 -- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fde6c0b7be0 msgr2=0x7fde6c1548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fde6c0b7be0 0x7fde6c1548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 -- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6c15e790 msgr2=0x7fde6c15d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6c15e790 0x7fde6c15d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 -- 172.21.15.46:0/588110401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c154e30 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde72637640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6c15e790 0x7fde6c15d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde71635640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 0x7fde6c15d240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c410 tx=0x7fde6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.698+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde60017070 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.699+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde60005ce0 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.699+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6c0c8f70 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.699+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde600072f0 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.699+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6c0c9530 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.699+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6c0689e0 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.700+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde6001a070 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.701+0000 7fde6a7fc640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fde5806c8f0 0x7fde5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.701+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fde6008b180 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.701+0000 7fde71e36640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fde5806c8f0 0x7fde5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.701+0000 7fde71e36640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fde5806c8f0 0x7fde5806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fde64036000 tx=0x7fde64030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.705+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde60058c10 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.888+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fde6c0c9730 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.888+0000 7fde6a7fc640 1 -- 172.21.15.46:0/588110401 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fde6005cf10 con 0x7fde6c0d1cf0 2024-04-15T15:15:49.890 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:15:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fde5806c8f0 msgr2=0x7fde5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fde5806c8f0 0x7fde5806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fde64036000 tx=0x7fde64030000 comp rx=0 tx=0).stop 2024-04-15T15:15:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 msgr2=0x7fde6c15d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 0x7fde6c15d240 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde6000c410 tx=0x7fde6000c8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 shutdown_connections 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fde5806c8f0 0x7fde5806edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde6c15e790 0x7fde6c15d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde6c0d1cf0 0x7fde6c15d240 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 --2- 172.21.15.46:0/588110401 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fde6c0b7be0 0x7fde6c1548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 >> 172.21.15.46:0/588110401 conn(0x7fde6c067b40 msgr2=0x7fde6c0cb8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 shutdown_connections 2024-04-15T15:15:49.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:49.890+0000 7fde7409a640 1 -- 172.21.15.46:0/588110401 wait complete. 2024-04-15T15:15:49.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:50.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.008+0000 7f712404e640 1 Processor -- start 2024-04-15T15:15:50.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.008+0000 7f712404e640 1 -- start start 2024-04-15T15:15:50.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f712404e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c15e700 0x7f711c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f712404e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f711c0d3270 0x7f711c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f712404e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711c0b47a0 con 0x7f711c0d2360 2024-04-15T15:15:50.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f712404e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f711c0b53b0 con 0x7f711c0d3270 2024-04-15T15:15:50.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f712404e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f711c0b4a80 con 0x7f711c15e700 2024-04-15T15:15:50.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f7121dea640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c15e700 0x7f711c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f71225eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f711c0d3270 0x7f711c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f71215e9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f711c0d2360 0x7f711c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:36316/0 (socket says 172.21.15.46:36316) 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f71215e9640 1 -- 172.21.15.46:0/372142157 learned_addr learned my addr 172.21.15.46:0/372142157 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f7121dea640 1 -- 172.21.15.46:0/372142157 >> v2:172.21.15.78:3300/0 conn(0x7f711c0d3270 msgr2=0x7f711c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f7121dea640 1 --2- 172.21.15.46:0/372142157 >> v2:172.21.15.78:3300/0 conn(0x7f711c0d3270 0x7f711c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f7121dea640 1 -- 172.21.15.46:0/372142157 >> v1:172.21.15.78:6789/0 conn(0x7f711c0d2360 legacy=0x7f711c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.009+0000 7f7121dea640 1 -- 172.21.15.46:0/372142157 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c0b4d60 con 0x7f711c15e700 2024-04-15T15:15:50.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.010+0000 7f7121dea640 1 --2- 172.21.15.46:0/372142157 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c15e700 0x7f711c0b5840 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7118009790 tx=0x7f711802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c164a56ee7c2b44d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.010+0000 7f7120de8640 1 -- 172.21.15.46:0/372142157 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7118035020 con 0x7f711c15e700 2024-04-15T15:15:50.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.010+0000 7f7120de8640 1 -- 172.21.15.46:0/372142157 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7118031890 con 0x7f711c15e700 2024-04-15T15:15:50.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.010+0000 7f7120de8640 1 -- 172.21.15.46:0/372142157 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7118031c60 con 0x7f711c15e700 2024-04-15T15:15:50.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 -- 172.21.15.46:0/372142157 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c15e700 msgr2=0x7f711c0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 --2- 172.21.15.46:0/372142157 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c15e700 0x7f711c0b5840 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7118009790 tx=0x7f711802b930 comp rx=0 tx=0).stop 2024-04-15T15:15:50.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 -- 172.21.15.46:0/372142157 shutdown_connections 2024-04-15T15:15:50.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 --2- 172.21.15.46:0/372142157 >> v2:172.21.15.78:3300/0 conn(0x7f711c0d3270 0x7f711c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 --2- 172.21.15.46:0/372142157 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c15e700 0x7f711c0b5840 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 -- 172.21.15.46:0/372142157 >> 172.21.15.46:0/372142157 conn(0x7f711c08bf60 msgr2=0x7f711c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 -- 172.21.15.46:0/372142157 shutdown_connections 2024-04-15T15:15:50.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.011+0000 7f712404e640 1 -- 172.21.15.46:0/372142157 wait complete. 2024-04-15T15:15:50.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.012+0000 7f712404e640 1 Processor -- start 2024-04-15T15:15:50.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.012+0000 7f712404e640 1 -- start start 2024-04-15T15:15:50.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f712404e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 0x7f711c14ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f712404e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c0d3270 0x7f711c0b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f712404e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f711c15e700 0x7f711c0b8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f712404e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f711c0b5d80 con 0x7f711c0d2360 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f712404e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f711c0b4450 con 0x7f711c15e700 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f712404e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f711c0d1d40 con 0x7f711c0d3270 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 0x7f711c14ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 0x7f711c14ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60252/0 (socket says 172.21.15.46:60252) 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 -- 172.21.15.46:0/1855136248 learned_addr learned my addr 172.21.15.46:0/1855136248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f71215e9640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c0d3270 0x7f711c0b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f71225eb640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f711c15e700 0x7f711c0b8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 -- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c0d3270 msgr2=0x7f711c0b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c0d3270 0x7f711c0b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 -- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f711c15e700 msgr2=0x7f711c0b8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f711c15e700 0x7f711c0b8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f7121dea640 1 -- 172.21.15.46:0/1855136248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c0b1690 con 0x7f711c0d2360 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.013+0000 7f71225eb640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f711c15e700 0x7f711c0b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.014+0000 7f7121dea640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 0x7f711c14ea30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f71180098b0 tx=0x7f71180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.014+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7118035040 con 0x7f711c0d2360 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.014+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711803e070 con 0x7f711c0d2360 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.014+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711c0b1970 con 0x7f711c0d2360 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.014+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7118005990 con 0x7f711c0d2360 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.014+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711c15fdb0 con 0x7f711c0d2360 2024-04-15T15:15:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.015+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711c0689e0 con 0x7f711c0d2360 2024-04-15T15:15:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.015+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7118038020 con 0x7f711c0d2360 2024-04-15T15:15:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.016+0000 7f71127fc640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f710006c8f0 0x7f710006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.016+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f71180adc50 con 0x7f711c0d2360 2024-04-15T15:15:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.016+0000 7f71215e9640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f710006c8f0 0x7f710006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.016+0000 7f71215e9640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f710006c8f0 0x7f710006edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f711c15ad90 tx=0x7f710c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.020+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71180b3020 con 0x7f711c0d2360 2024-04-15T15:15:50.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.205+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f711c0ba500 con 0x7f711c0d2360 2024-04-15T15:15:50.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.205+0000 7f71127fc640 1 -- 172.21.15.46:0/1855136248 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f711802d040 con 0x7f711c0d2360 2024-04-15T15:15:50.207 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:15:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f710006c8f0 msgr2=0x7f710006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f710006c8f0 0x7f710006edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f711c15ad90 tx=0x7f710c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 msgr2=0x7f711c14ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 0x7f711c14ea30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f71180098b0 tx=0x7f71180046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 shutdown_connections 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f710006c8f0 0x7f710006edb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f711c15e700 0x7f711c0b8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f711c0d3270 0x7f711c0b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 --2- 172.21.15.46:0/1855136248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f711c0d2360 0x7f711c14ea30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 >> 172.21.15.46:0/1855136248 conn(0x7f711c08bf60 msgr2=0x7f711c0c97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 shutdown_connections 2024-04-15T15:15:50.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.207+0000 7f712404e640 1 -- 172.21.15.46:0/1855136248 wait complete. 2024-04-15T15:15:50.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:50.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.325+0000 7fb3725c8640 1 Processor -- start 2024-04-15T15:15:50.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.325+0000 7fb3725c8640 1 -- start start 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb3725c8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb36c09a9d0 0x7fb36c09adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb3725c8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb3725c8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36c09b2f0 con 0x7fb36c15ebb0 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb3725c8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c0cd8c0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb3725c8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c0ced60 con 0x7fb36c09b780 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb36bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb36c09a9d0 0x7fb36c09adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb36bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb36c09a9d0 0x7fb36c09adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60256/0 (socket says 172.21.15.46:60256) 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.326+0000 7fb36bfff640 1 -- 172.21.15.46:0/2018851381 learned_addr learned my addr 172.21.15.46:0/2018851381 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.327+0000 7fb370b65640 1 --2- 172.21.15.46:0/2018851381 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.327+0000 7fb36bfff640 1 -- 172.21.15.46:0/2018851381 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 msgr2=0x7fb36c0b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.327+0000 7fb36bfff640 1 --2- 172.21.15.46:0/2018851381 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.327+0000 7fb36bfff640 1 -- 172.21.15.46:0/2018851381 >> v1:172.21.15.78:6789/0 conn(0x7fb36c15ebb0 legacy=0x7fb36c06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.327+0000 7fb36bfff640 1 -- 172.21.15.46:0/2018851381 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c0cccb0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.327+0000 7fb36bfff640 1 --2- 172.21.15.46:0/2018851381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c09adb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009870 tx=0x7fb35c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dc3810605c1534 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb36affd640 1 -- 172.21.15.46:0/2018851381 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c035020 con 0x7fb36c09a9d0 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb36affd640 1 -- 172.21.15.46:0/2018851381 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb35c031640 con 0x7fb36c09a9d0 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb36affd640 1 -- 172.21.15.46:0/2018851381 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c02d040 con 0x7fb36c09a9d0 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 -- 172.21.15.46:0/2018851381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 msgr2=0x7fb36c09adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 --2- 172.21.15.46:0/2018851381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c09adb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb35c009870 tx=0x7fb35c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 -- 172.21.15.46:0/2018851381 shutdown_connections 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 --2- 172.21.15.46:0/2018851381 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 --2- 172.21.15.46:0/2018851381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c09adb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 -- 172.21.15.46:0/2018851381 >> 172.21.15.46:0/2018851381 conn(0x7fb36c0b3fb0 msgr2=0x7fb36c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 -- 172.21.15.46:0/2018851381 shutdown_connections 2024-04-15T15:15:50.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.328+0000 7fb3725c8640 1 -- 172.21.15.46:0/2018851381 wait complete. 2024-04-15T15:15:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.329+0000 7fb3725c8640 1 Processor -- start 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 -- start start 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb36c0b3940 0x7fb36c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c0cd550 con 0x7fb36c09a9d0 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c0cd270 con 0x7fb36c0b3940 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb3725c8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c0b5310 con 0x7fb36c09b780 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb36bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb36bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60264/0 (socket says 172.21.15.46:60264) 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.330+0000 7fb36bfff640 1 -- 172.21.15.46:0/4112765183 learned_addr learned my addr 172.21.15.46:0/4112765183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36b7fe640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb370b65640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb36c0b3940 0x7fb36c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36bfff640 1 -- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 msgr2=0x7fb36c0b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36bfff640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36bfff640 1 -- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb36c0b3940 msgr2=0x7fb36c155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36bfff640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb36c0b3940 0x7fb36c155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36bfff640 1 -- 172.21.15.46:0/4112765183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c156ca0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb370b65640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb36c0b3940 0x7fb36c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb36bfff640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c153e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb35c02b930 tx=0x7fb35c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c040070 con 0x7fb36c09a9d0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb35c0399f0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c156f80 con 0x7fb36c09a9d0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c035040 con 0x7fb36c09a9d0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.331+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c156470 con 0x7fb36c09a9d0 2024-04-15T15:15:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.332+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c0b3d50 con 0x7fb36c09a9d0 2024-04-15T15:15:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.333+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb35c04b060 con 0x7fb36c09a9d0 2024-04-15T15:15:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.333+0000 7fb368ff9640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb35806c8f0 0x7fb35806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.333+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb35c0ad7d0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.333+0000 7fb36b7fe640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb35806c8f0 0x7fb35806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.334+0000 7fb36b7fe640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb35806c8f0 0x7fb35806edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb360006fd0 tx=0x7fb360008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.338+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb36c0b3d50 con 0x7fb36c09a9d0 2024-04-15T15:15:50.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.536+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb36c0690f0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.537+0000 7fb368ff9640 1 -- 172.21.15.46:0/4112765183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7fb35c07b8d0 con 0x7fb36c09a9d0 2024-04-15T15:15:50.538 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.538+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb35806c8f0 msgr2=0x7fb35806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.538+0000 7fb3725c8640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb35806c8f0 0x7fb35806edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb360006fd0 tx=0x7fb360008040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 msgr2=0x7fb36c153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c153e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb35c02b930 tx=0x7fb35c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 shutdown_connections 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb35806c8f0 0x7fb35806edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb36c0b3940 0x7fb36c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb36c09b780 0x7fb36c0b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 --2- 172.21.15.46:0/4112765183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb36c09a9d0 0x7fb36c153e50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 >> 172.21.15.46:0/4112765183 conn(0x7fb36c0b3fb0 msgr2=0x7fb36c0cef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 shutdown_connections 2024-04-15T15:15:50.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.539+0000 7fb3725c8640 1 -- 172.21.15.46:0/4112765183 wait complete. 2024-04-15T15:15:50.551 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:15:50.553 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:50.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.714+0000 7fb5b73ba640 1 Processor -- start 2024-04-15T15:15:50.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.715+0000 7fb5b73ba640 1 -- start start 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.715+0000 7fb5b73ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b00d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.715+0000 7fb5b73ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5b00d1610 0x7fb5b00d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.715+0000 7fb5b73ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b00d2480 con 0x7fb5b00cfd20 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.715+0000 7fb5b73ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb5b00d2db0 con 0x7fb5b00d1610 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.715+0000 7fb5b73ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b00d2760 con 0x7fb5b015e790 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5b00d1610 0x7fb5b00d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5b00d1610 0x7fb5b00d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60276/0 (socket says 172.21.15.46:60276) 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 -- 172.21.15.46:0/4257007512 learned_addr learned my addr 172.21.15.46:0/4257007512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5156640 1 --2- 172.21.15.46:0/4257007512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b00d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 -- 172.21.15.46:0/4257007512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 msgr2=0x7fb5b00d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 --2- 172.21.15.46:0/4257007512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b00d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 -- 172.21.15.46:0/4257007512 >> v1:172.21.15.78:6789/0 conn(0x7fb5b00cfd20 legacy=0x7fb5b00d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.716+0000 7fb5b5957640 1 -- 172.21.15.46:0/4257007512 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b00d2a40 con 0x7fb5b00d1610 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b5957640 1 --2- 172.21.15.46:0/4257007512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00d1610 0x7fb5b00d0160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009e50 tx=0x7fb5a802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960a4adb234a19c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5a7fff640 1 -- 172.21.15.46:0/4257007512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a802cb00 con 0x7fb5b00d1610 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5a7fff640 1 -- 172.21.15.46:0/4257007512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a802cca0 con 0x7fb5b00d1610 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5a7fff640 1 -- 172.21.15.46:0/4257007512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a8034610 con 0x7fb5b00d1610 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b73ba640 1 -- 172.21.15.46:0/4257007512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00d1610 msgr2=0x7fb5b00d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/4257007512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00d1610 0x7fb5b00d0160 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009e50 tx=0x7fb5a802bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b73ba640 1 -- 172.21.15.46:0/4257007512 shutdown_connections 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/4257007512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00d1610 0x7fb5b00d0160 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/4257007512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b00d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.717+0000 7fb5b73ba640 1 -- 172.21.15.46:0/4257007512 >> 172.21.15.46:0/4257007512 conn(0x7fb5b008bf60 msgr2=0x7fb5b00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.718+0000 7fb5b73ba640 1 -- 172.21.15.46:0/4257007512 shutdown_connections 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.718+0000 7fb5b73ba640 1 -- 172.21.15.46:0/4257007512 wait complete. 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 Processor -- start 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 -- start start 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 0x7fb5b00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5b00d1610 0x7fb5b014fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b006d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b015ecf0 con 0x7fb5b00cfd20 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b015eb70 con 0x7fb5b00d1610 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.719+0000 7fb5b73ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b00b74f0 con 0x7fb5b015e790 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 0x7fb5b00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 0x7fb5b00cc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60280/0 (socket says 172.21.15.46:60280) 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 -- 172.21.15.46:0/911866261 learned_addr learned my addr 172.21.15.46:0/911866261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5957640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b006d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b4955640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5b00d1610 0x7fb5b014fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 -- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 msgr2=0x7fb5b006d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b006d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 -- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5b00d1610 msgr2=0x7fb5b014fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5b00d1610 0x7fb5b014fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 -- 172.21.15.46:0/911866261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b00cb8a0 con 0x7fb5b00cfd20 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b5156640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 0x7fb5b00cc940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00e870 tx=0x7fb5ac00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac017070 con 0x7fb5b00cfd20 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ac00c940 con 0x7fb5b00cfd20 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00cd30 con 0x7fb5b00cfd20 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.720+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b00cbb80 con 0x7fb5b00cfd20 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.721+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b00cc140 con 0x7fb5b00cfd20 2024-04-15T15:15:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.721+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b00689e0 con 0x7fb5b00cfd20 2024-04-15T15:15:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.722+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb5ac01a070 con 0x7fb5b00cfd20 2024-04-15T15:15:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.722+0000 7fb5a5ffb640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb59406c8f0 0x7fb59406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.723+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb5ac08c0a0 con 0x7fb5b00cfd20 2024-04-15T15:15:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.723+0000 7fb5b4955640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb59406c8f0 0x7fb59406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.723+0000 7fb5b4955640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb59406c8f0 0x7fb59406edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0150270 tx=0x7fb5a0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.727+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb5ac05a180 con 0x7fb5b00cfd20 2024-04-15T15:15:50.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.912+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5b00d1ce0 con 0x7fb5b00cfd20 2024-04-15T15:15:50.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.913+0000 7fb5a5ffb640 1 -- 172.21.15.46:0/911866261 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fb5ac059a80 con 0x7fb5b00cfd20 2024-04-15T15:15:50.915 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:50.915 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:50.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb59406c8f0 msgr2=0x7fb59406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb59406c8f0 0x7fb59406edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0150270 tx=0x7fb5a0009040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 msgr2=0x7fb5b00cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 0x7fb5b00cc940 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00e870 tx=0x7fb5ac00ed30 comp rx=0 tx=0).stop 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 shutdown_connections 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb59406c8f0 0x7fb59406edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5b015e790 0x7fb5b006d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5b00d1610 0x7fb5b014fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 --2- 172.21.15.46:0/911866261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5b00cfd20 0x7fb5b00cc940 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 >> 172.21.15.46:0/911866261 conn(0x7fb5b008bf60 msgr2=0x7fb5b00d4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 shutdown_connections 2024-04-15T15:15:50.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:50.915+0000 7fb5b73ba640 1 -- 172.21.15.46:0/911866261 wait complete. 2024-04-15T15:15:50.928 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:50.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:51.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.034+0000 7fee88a74640 1 Processor -- start 2024-04-15T15:15:51.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.034+0000 7fee88a74640 1 -- start start 2024-04-15T15:15:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee88a74640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee8409a9d0 0x7fee8409adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee88a74640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fee8415ebb0 0x7fee8406cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee88a74640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee8409b2f0 con 0x7fee8409b780 2024-04-15T15:15:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee88a74640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fee840cd8c0 con 0x7fee8409a9d0 2024-04-15T15:15:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee88a74640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fee840ced60 con 0x7fee8415ebb0 2024-04-15T15:15:51.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee8259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee8409a9d0 0x7fee8409adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee8259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fee8409a9d0 0x7fee8409adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60286/0 (socket says 172.21.15.46:60286) 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.035+0000 7fee8259c640 1 -- 172.21.15.46:0/3449436823 learned_addr learned my addr 172.21.15.46:0/3449436823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee81d9b640 1 --2- 172.21.15.46:0/3449436823 >> v2:172.21.15.78:3300/0 conn(0x7fee8415ebb0 0x7fee8406cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee8259c640 1 -- 172.21.15.46:0/3449436823 >> v2:172.21.15.78:3300/0 conn(0x7fee8415ebb0 msgr2=0x7fee8406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee8259c640 1 --2- 172.21.15.46:0/3449436823 >> v2:172.21.15.78:3300/0 conn(0x7fee8415ebb0 0x7fee8406cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee8259c640 1 -- 172.21.15.46:0/3449436823 >> v1:172.21.15.78:6789/0 conn(0x7fee8409b780 legacy=0x7fee840b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee8259c640 1 -- 172.21.15.46:0/3449436823 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee840cccb0 con 0x7fee8409a9d0 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee81d9b640 1 --2- 172.21.15.46:0/3449436823 >> v2:172.21.15.78:3300/0 conn(0x7fee8415ebb0 0x7fee8406cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.036+0000 7fee8259c640 1 --2- 172.21.15.46:0/3449436823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee8409a9d0 0x7fee8409adb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fee840d0560 tx=0x7fee7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3b29ad9b71e91c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee8159a640 1 -- 172.21.15.46:0/3449436823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c035020 con 0x7fee8409a9d0 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee8159a640 1 -- 172.21.15.46:0/3449436823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7c031870 con 0x7fee8409a9d0 2024-04-15T15:15:51.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee8159a640 1 -- 172.21.15.46:0/3449436823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c031c40 con 0x7fee8409a9d0 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 -- 172.21.15.46:0/3449436823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee8409a9d0 msgr2=0x7fee8409adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 --2- 172.21.15.46:0/3449436823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee8409a9d0 0x7fee8409adb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fee840d0560 tx=0x7fee7c02b910 comp rx=0 tx=0).stop 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 -- 172.21.15.46:0/3449436823 shutdown_connections 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 --2- 172.21.15.46:0/3449436823 >> v2:172.21.15.78:3300/0 conn(0x7fee8415ebb0 0x7fee8406cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 --2- 172.21.15.46:0/3449436823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee8409a9d0 0x7fee8409adb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 -- 172.21.15.46:0/3449436823 >> 172.21.15.46:0/3449436823 conn(0x7fee840b3fb0 msgr2=0x7fee840b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 -- 172.21.15.46:0/3449436823 shutdown_connections 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.037+0000 7fee88a74640 1 -- 172.21.15.46:0/3449436823 wait complete. 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 Processor -- start 2024-04-15T15:15:51.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 -- start start 2024-04-15T15:15:51.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fee8409a9d0 0x7fee840b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8409b780 0x7fee841f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 0x7fee840c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee840cd550 con 0x7fee841f8760 2024-04-15T15:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fee840cd270 con 0x7fee8409b780 2024-04-15T15:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee88a74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fee8406d240 con 0x7fee8409a9d0 2024-04-15T15:15:51.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee82d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 0x7fee840c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee82d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 0x7fee840c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60298/0 (socket says 172.21.15.46:60298) 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.039+0000 7fee82d9d640 1 -- 172.21.15.46:0/3651252576 learned_addr learned my addr 172.21.15.46:0/3651252576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee81d9b640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8409b780 0x7fee841f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee8259c640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fee8409a9d0 0x7fee840b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee82d9d640 1 -- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fee8409a9d0 msgr2=0x7fee840b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee82d9d640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fee8409a9d0 0x7fee840b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee82d9d640 1 -- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8409b780 msgr2=0x7fee841f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee82d9d640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8409b780 0x7fee841f19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee82d9d640 1 -- 172.21.15.46:0/3651252576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee840b7a80 con 0x7fee841f8760 2024-04-15T15:15:51.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee82d9d640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 0x7fee840c79f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee7400c780 tx=0x7fee7400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74004b60 con 0x7fee841f8760 2024-04-15T15:15:51.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee74004d00 con 0x7fee841f8760 2024-04-15T15:15:51.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7400aa90 con 0x7fee841f8760 2024-04-15T15:15:51.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee841f47b0 con 0x7fee841f8760 2024-04-15T15:15:51.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.040+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee841f4d70 con 0x7fee841f8760 2024-04-15T15:15:51.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.041+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee840690f0 con 0x7fee841f8760 2024-04-15T15:15:51.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.042+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fee7400ac30 con 0x7fee841f8760 2024-04-15T15:15:51.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.042+0000 7fee7affd640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fee6406c8f0 0x7fee6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.042+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fee7408c120 con 0x7fee841f8760 2024-04-15T15:15:51.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.042+0000 7fee8259c640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fee6406c8f0 0x7fee6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.043+0000 7fee8259c640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fee6406c8f0 0x7fee6406edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fee7c036000 tx=0x7fee7c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.046+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee74020070 con 0x7fee841f8760 2024-04-15T15:15:51.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.229+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fee840b7cb0 con 0x7fee841f8760 2024-04-15T15:15:51.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.229+0000 7fee7affd640 1 -- 172.21.15.46:0/3651252576 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fee840b7a80 con 0x7fee841f8760 2024-04-15T15:15:51.231 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:15:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fee6406c8f0 msgr2=0x7fee6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fee6406c8f0 0x7fee6406edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fee7c036000 tx=0x7fee7c008030 comp rx=0 tx=0).stop 2024-04-15T15:15:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 msgr2=0x7fee840c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 0x7fee840c79f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fee7400c780 tx=0x7fee7400cc40 comp rx=0 tx=0).stop 2024-04-15T15:15:51.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 shutdown_connections 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fee6406c8f0 0x7fee6406edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee841f8760 0x7fee840c79f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee8409b780 0x7fee841f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 --2- 172.21.15.46:0/3651252576 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fee8409a9d0 0x7fee840b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 >> 172.21.15.46:0/3651252576 conn(0x7fee840b3fb0 msgr2=0x7fee84070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 shutdown_connections 2024-04-15T15:15:51.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.231+0000 7fee88a74640 1 -- 172.21.15.46:0/3651252576 wait complete. 2024-04-15T15:15:51.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.351+0000 7f83e2f15640 1 Processor -- start 2024-04-15T15:15:51.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.351+0000 7f83e2f15640 1 -- start start 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e2f15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83dc0d1610 0x7f83dc0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e2f15640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83dc0d2480 con 0x7f83dc15e790 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e2f15640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83dc0d2760 con 0x7f83dc0d3240 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e2f15640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f83dc0d2db0 con 0x7f83dc0d1610 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e14b2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83dc0d1610 0x7f83dc0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e0cb1640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f83dc15e790 0x7f83dc0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41512/0 (socket says 172.21.15.46:41512) 2024-04-15T15:15:51.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e0cb1640 1 -- 172.21.15.46:0/1334232435 learned_addr learned my addr 172.21.15.46:0/1334232435 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e14b2640 1 -- 172.21.15.46:0/1334232435 >> v1:172.21.15.78:6789/0 conn(0x7f83dc0d3240 legacy=0x7f83dc0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e14b2640 1 -- 172.21.15.46:0/1334232435 >> v1:172.21.15.46:6789/0 conn(0x7f83dc15e790 legacy=0x7f83dc0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83e14b2640 1 -- 172.21.15.46:0/1334232435 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc0d2a40 con 0x7f83dc0d1610 2024-04-15T15:15:51.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.352+0000 7f83d37fe640 1 -- 172.21.15.46:0/1334232435 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557964400 0 0) 0x7f83dc0d2480 con 0x7f83dc15e790 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.353+0000 7f83e14b2640 1 --2- 172.21.15.46:0/1334232435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d1610 0x7f83dc0b74f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0d2f30 tx=0x7f83d402bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e64fbe5166fea server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.353+0000 7f83d37fe640 1 -- 172.21.15.46:0/1334232435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4039070 con 0x7f83dc0d1610 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.353+0000 7f83d37fe640 1 -- 172.21.15.46:0/1334232435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d402ccd0 con 0x7f83dc0d1610 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.353+0000 7f83d37fe640 1 -- 172.21.15.46:0/1334232435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4034660 con 0x7f83dc0d1610 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 -- 172.21.15.46:0/1334232435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d1610 msgr2=0x7f83dc0b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 --2- 172.21.15.46:0/1334232435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d1610 0x7f83dc0b74f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f83dc0d2f30 tx=0x7f83d402bf20 comp rx=0 tx=0).stop 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 -- 172.21.15.46:0/1334232435 shutdown_connections 2024-04-15T15:15:51.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 --2- 172.21.15.46:0/1334232435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d1610 0x7f83dc0b74f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 -- 172.21.15.46:0/1334232435 >> 172.21.15.46:0/1334232435 conn(0x7f83dc08bf60 msgr2=0x7f83dc0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 -- 172.21.15.46:0/1334232435 shutdown_connections 2024-04-15T15:15:51.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.354+0000 7f83e2f15640 1 -- 172.21.15.46:0/1334232435 wait complete. 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.355+0000 7f83e2f15640 1 Processor -- start 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.355+0000 7f83e2f15640 1 -- start start 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e2f15640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83dc0d1610 0x7f83dc0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e2f15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 0x7f83dc0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e2f15640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83dc15e790 0x7f83dc0cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e2f15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc0d5180 con 0x7f83dc0d3240 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e2f15640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc0d0430 con 0x7f83dc15e790 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e2f15640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f83dc0d5300 con 0x7f83dc0d1610 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 0x7f83dc0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 0x7f83dc0cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60310/0 (socket says 172.21.15.46:60310) 2024-04-15T15:15:51.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 -- 172.21.15.46:0/757555392 learned_addr learned my addr 172.21.15.46:0/757555392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e14b2640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83dc15e790 0x7f83dc0cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e0cb1640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83dc0d1610 0x7f83dc0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 -- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83dc0d1610 msgr2=0x7f83dc0ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83dc0d1610 0x7f83dc0ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 -- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83dc15e790 msgr2=0x7f83dc0cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83dc15e790 0x7f83dc0cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 -- 172.21.15.46:0/757555392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83dc0c9c20 con 0x7f83dc0d3240 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83e14b2640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83dc15e790 0x7f83dc0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.356+0000 7f83d3fff640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 0x7f83dc0cd780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00c450 tx=0x7f83cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.357+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc0052d0 con 0x7f83dc0d3240 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.357+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83cc005470 con 0x7f83dc0d3240 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.357+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83dc0cabb0 con 0x7f83dc0d3240 2024-04-15T15:15:51.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.357+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc012870 con 0x7f83dc0d3240 2024-04-15T15:15:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.357+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc0c8e60 con 0x7f83dc0d3240 2024-04-15T15:15:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.358+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83dc0689e0 con 0x7f83dc0d3240 2024-04-15T15:15:51.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.358+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f83cc012a10 con 0x7f83dc0d3240 2024-04-15T15:15:51.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.359+0000 7f83d17fa640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f83c806c8f0 0x7f83c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.359+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f83cc08bb30 con 0x7f83dc0d3240 2024-04-15T15:15:51.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.359+0000 7f83e0cb1640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f83c806c8f0 0x7f83c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.359+0000 7f83e0cb1640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f83c806c8f0 0x7f83c806edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000f60 tx=0x7f83d8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.363+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83cc059530 con 0x7f83dc0d3240 2024-04-15T15:15:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.547+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f83dc0d1ce0 con 0x7f83dc0d3240 2024-04-15T15:15:51.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.547+0000 7f83d17fa640 1 -- 172.21.15.46:0/757555392 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f83cc05d830 con 0x7f83dc0d3240 2024-04-15T15:15:51.549 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:15:51.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.549+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f83c806c8f0 msgr2=0x7f83c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.549+0000 7f83e2f15640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f83c806c8f0 0x7f83c806edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000f60 tx=0x7f83d8009040 comp rx=0 tx=0).stop 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.549+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 msgr2=0x7f83dc0cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.549+0000 7f83e2f15640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 0x7f83dc0cd780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f83cc00c450 tx=0x7f83cc00c910 comp rx=0 tx=0).stop 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 shutdown_connections 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f83c806c8f0 0x7f83c806edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83dc15e790 0x7f83dc0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83dc0d3240 0x7f83dc0cd780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 --2- 172.21.15.46:0/757555392 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83dc0d1610 0x7f83dc0ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 >> 172.21.15.46:0/757555392 conn(0x7f83dc08bf60 msgr2=0x7f83dc06e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 shutdown_connections 2024-04-15T15:15:51.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.550+0000 7f83e2f15640 1 -- 172.21.15.46:0/757555392 wait complete. 2024-04-15T15:15:51.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.668+0000 7fa3ac27b640 1 Processor -- start 2024-04-15T15:15:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.668+0000 7fa3ac27b640 1 -- start start 2024-04-15T15:15:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3ac27b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3a40b5ad0 0x7fa3a40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3ac27b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a4067e30 con 0x7fa3a415e830 2024-04-15T15:15:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3ac27b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a40b3c10 con 0x7fa3a415ec10 2024-04-15T15:15:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3ac27b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a40b3ef0 con 0x7fa3a40b5ad0 2024-04-15T15:15:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3aa017640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa3a415e830 0x7fa3a4067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41514/0 (socket says 172.21.15.46:41514) 2024-04-15T15:15:51.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3aa017640 1 -- 172.21.15.46:0/3330937178 learned_addr learned my addr 172.21.15.46:0/3330937178 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3aa818640 1 --2- 172.21.15.46:0/3330937178 >> v2:172.21.15.78:3300/0 conn(0x7fa3a40b5ad0 0x7fa3a40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404561426 0 0) 0x7fa3a4067e30 con 0x7fa3a415e830 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa390003520 con 0x7fa3a415e830 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3016817726 0 0) 0x7fa390003520 con 0x7fa3a415e830 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 >> v2:172.21.15.78:3300/0 conn(0x7fa3a40b5ad0 msgr2=0x7fa3a40b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 --2- 172.21.15.46:0/3330937178 >> v2:172.21.15.78:3300/0 conn(0x7fa3a40b5ad0 0x7fa3a40b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 >> v1:172.21.15.78:6789/0 conn(0x7fa3a415ec10 legacy=0x7fa3a40b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a40b41d0 con 0x7fa3a415e830 2024-04-15T15:15:51.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.669+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fa3a0001670 con 0x7fa3a415e830 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 >> v1:172.21.15.46:6789/0 conn(0x7fa3a415e830 legacy=0x7fa3a4067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3900039b0 0x7fa390003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa390004800 0x7fa390004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3900051c0 0x7fa39000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a40b3ef0 con 0x7fa390004800 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa39000c160 con 0x7fa3900039b0 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa39000c360 con 0x7fa3900051c0 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9816640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa390004800 0x7fa390004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3aa017640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3900039b0 0x7fa390003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3aa818640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3900051c0 0x7fa39000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9816640 1 -- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3900051c0 msgr2=0x7fa39000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9816640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3900051c0 0x7fa39000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9816640 1 -- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3900039b0 msgr2=0x7fa390003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9816640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3900039b0 0x7fa390003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3aa818640 1 -- 172.21.15.46:0/3330937178 reap_dead start 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.670+0000 7fa3a9816640 1 -- 172.21.15.46:0/3330937178 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa390003520 con 0x7fa390004800 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.671+0000 7fa3a9816640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa390004800 0x7fa390004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4070ed0 tx=0x7fa390005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236f48a0b055c78a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.671+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394001370 con 0x7fa390004800 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.671+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3940052d0 con 0x7fa390004800 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.671+0000 7fa3a9015640 1 -- 172.21.15.46:0/3330937178 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394003830 con 0x7fa390004800 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.671+0000 7fa3ac27b640 1 -- 172.21.15.46:0/3330937178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa390004800 msgr2=0x7fa390004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.671+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa390004800 0x7fa390004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4070ed0 tx=0x7fa390005830 comp rx=0 tx=0).stop 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.672+0000 7fa3ac27b640 1 -- 172.21.15.46:0/3330937178 shutdown_connections 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.672+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/3330937178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa390004800 0x7fa390004c80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.672+0000 7fa3ac27b640 1 -- 172.21.15.46:0/3330937178 >> 172.21.15.46:0/3330937178 conn(0x7fa3a4073360 msgr2=0x7fa3a406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.672+0000 7fa3ac27b640 1 -- 172.21.15.46:0/3330937178 shutdown_connections 2024-04-15T15:15:51.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.672+0000 7fa3ac27b640 1 -- 172.21.15.46:0/3330937178 wait complete. 2024-04-15T15:15:51.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 Processor -- start 2024-04-15T15:15:51.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 -- start start 2024-04-15T15:15:51.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa390004800 0x7fa3a4153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 0x7fa3a41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a41434d0 0x7fa3a415ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a415ff60 con 0x7fa3a41555e0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a40b4790 con 0x7fa3a41434d0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.673+0000 7fa3ac27b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a41600e0 con 0x7fa390004800 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 0x7fa3a41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 0x7fa3a41592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60336/0 (socket says 172.21.15.46:60336) 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 -- 172.21.15.46:0/178013727 learned_addr learned my addr 172.21.15.46:0/178013727 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3aa017640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa390004800 0x7fa3a4153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3aa818640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a41434d0 0x7fa3a415ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 -- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa390004800 msgr2=0x7fa3a4153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa390004800 0x7fa3a4153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 -- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a41434d0 msgr2=0x7fa3a415ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a41434d0 0x7fa3a415ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 -- 172.21.15.46:0/178013727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a4154540 con 0x7fa3a41555e0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3a9816640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 0x7fa3a41592d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa394005040 tx=0x7fa394001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394041020 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa394005480 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a4154820 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394001370 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.674+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a4154de0 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.675+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a40689e0 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.676+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa394044020 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.676+0000 7fa39affd640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa38806c8f0 0x7fa38806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.676+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa3940ad7a0 con 0x7fa3a41555e0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.676+0000 7fa3aa017640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa38806c8f0 0x7fa38806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.677+0000 7fa3aa017640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa38806c8f0 0x7fa38806edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0001050 tx=0x7fa3a00078f0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.680+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa39404b030 con 0x7fa3a41555e0 2024-04-15T15:15:51.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.864+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa3a406ff10 con 0x7fa3a41555e0 2024-04-15T15:15:51.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.865+0000 7fa39affd640 1 -- 172.21.15.46:0/178013727 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fa39407b240 con 0x7fa3a41555e0 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.866+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa38806c8f0 msgr2=0x7fa38806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.866+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa38806c8f0 0x7fa38806edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0001050 tx=0x7fa3a00078f0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.866+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 msgr2=0x7fa3a41592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.866+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 0x7fa3a41592d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa394005040 tx=0x7fa394001120 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 shutdown_connections 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa38806c8f0 0x7fa38806edb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3a41434d0 0x7fa3a415ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3a41555e0 0x7fa3a41592d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 --2- 172.21.15.46:0/178013727 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa390004800 0x7fa3a4153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 >> 172.21.15.46:0/178013727 conn(0x7fa3a4073360 msgr2=0x7fa3a40b1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 shutdown_connections 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.867+0000 7fa3ac27b640 1 -- 172.21.15.46:0/178013727 wait complete. 2024-04-15T15:15:51.880 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.986+0000 7fd6b17f2640 1 Processor -- start 2024-04-15T15:15:51.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.986+0000 7fd6b17f2640 1 -- start start 2024-04-15T15:15:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6b17f2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6ac071d10 0x7fd6ac0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6b17f2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ac15db40 con 0x7fd6ac072d80 2024-04-15T15:15:51.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6b17f2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ac072170 con 0x7fd6ac0d0bd0 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6b17f2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ac0d1040 con 0x7fd6ac071d10 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6aa7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd6ac071d10 0x7fd6ac0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6aaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6ac072d80 0x7fd6ac0af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41528/0 (socket says 172.21.15.46:41528) 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.987+0000 7fd6aaffd640 1 -- 172.21.15.46:0/1960116343 learned_addr learned my addr 172.21.15.46:0/1960116343 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.988+0000 7fd6aa7fc640 1 -- 172.21.15.46:0/1960116343 >> v1:172.21.15.78:6789/0 conn(0x7fd6ac0d0bd0 legacy=0x7fd6ac06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.988+0000 7fd6aa7fc640 1 -- 172.21.15.46:0/1960116343 >> v1:172.21.15.46:6789/0 conn(0x7fd6ac072d80 legacy=0x7fd6ac0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.988+0000 7fd6aa7fc640 1 -- 172.21.15.46:0/1960116343 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac15cfc0 con 0x7fd6ac071d10 2024-04-15T15:15:51.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.988+0000 7fd6a9ffb640 1 -- 172.21.15.46:0/1960116343 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3377053815 0 0) 0x7fd6ac15db40 con 0x7fd6ac072d80 2024-04-15T15:15:51.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.988+0000 7fd6aa7fc640 1 --2- 172.21.15.46:0/1960116343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac0d0690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd6940097a0 tx=0x7fd69402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bbb019d5614c1b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6a9ffb640 1 -- 172.21.15.46:0/1960116343 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6940315a0 con 0x7fd6ac071d10 2024-04-15T15:15:51.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6a9ffb640 1 -- 172.21.15.46:0/1960116343 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd694037070 con 0x7fd6ac071d10 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6a9ffb640 1 -- 172.21.15.46:0/1960116343 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd694031960 con 0x7fd6ac071d10 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 -- 172.21.15.46:0/1960116343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 msgr2=0x7fd6ac0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/1960116343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac0d0690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd6940097a0 tx=0x7fd69402b840 comp rx=0 tx=0).stop 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 -- 172.21.15.46:0/1960116343 shutdown_connections 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/1960116343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac0d0690 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 -- 172.21.15.46:0/1960116343 >> 172.21.15.46:0/1960116343 conn(0x7fd6ac06be00 msgr2=0x7fd6ac06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 -- 172.21.15.46:0/1960116343 shutdown_connections 2024-04-15T15:15:51.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.989+0000 7fd6b17f2640 1 -- 172.21.15.46:0/1960116343 wait complete. 2024-04-15T15:15:51.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.990+0000 7fd6b17f2640 1 Processor -- start 2024-04-15T15:15:51.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.990+0000 7fd6b17f2640 1 -- start start 2024-04-15T15:15:51.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6b17f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6b17f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd6ac072d80 0x7fd6ac15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6b17f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6ac0d0bd0 0x7fd6ac150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6b17f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac15d860 con 0x7fd6ac071d10 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6b17f2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac15d580 con 0x7fd6ac0d0bd0 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6b17f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac0afaf0 con 0x7fd6ac072d80 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6aaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6aaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60364/0 (socket says 172.21.15.46:60364) 2024-04-15T15:15:51.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6aaffd640 1 -- 172.21.15.46:0/2917642867 learned_addr learned my addr 172.21.15.46:0/2917642867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6aa7fc640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd6ac072d80 0x7fd6ac15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.991+0000 7fd6ab7fe640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6ac0d0bd0 0x7fd6ac150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aaffd640 1 -- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd6ac072d80 msgr2=0x7fd6ac15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aaffd640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd6ac072d80 0x7fd6ac15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aaffd640 1 -- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6ac0d0bd0 msgr2=0x7fd6ac150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aaffd640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6ac0d0bd0 0x7fd6ac150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aaffd640 1 -- 172.21.15.46:0/2917642867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac0cd990 con 0x7fd6ac071d10 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aa7fc640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd6ac072d80 0x7fd6ac15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6ab7fe640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6ac0d0bd0 0x7fd6ac150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6aaffd640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac150c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c4d0 tx=0x7fd6a400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4018070 con 0x7fd6ac071d10 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a4007500 con 0x7fd6ac071d10 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac0cdc30 con 0x7fd6ac071d10 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4004a20 con 0x7fd6ac071d10 2024-04-15T15:15:51.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.992+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac157250 con 0x7fd6ac071d10 2024-04-15T15:15:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.993+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ac0753a0 con 0x7fd6ac071d10 2024-04-15T15:15:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.993+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6a401b070 con 0x7fd6ac071d10 2024-04-15T15:15:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.994+0000 7fd6a37fe640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd68c06c8f0 0x7fd68c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.994+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd6a408b190 con 0x7fd6ac071d10 2024-04-15T15:15:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.994+0000 7fd6aa7fc640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd68c06c8f0 0x7fd68c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.995+0000 7fd6aa7fc640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd68c06c8f0 0x7fd68c06edb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd694036000 tx=0x7fd694008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:51.998+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6ac0753a0 con 0x7fd6ac071d10 2024-04-15T15:15:52.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.182+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd6ac0689e0 con 0x7fd6ac071d10 2024-04-15T15:15:52.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.183+0000 7fd6a37fe640 1 -- 172.21.15.46:0/2917642867 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fd6a4059280 con 0x7fd6ac071d10 2024-04-15T15:15:52.185 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:15:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd68c06c8f0 msgr2=0x7fd68c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd68c06c8f0 0x7fd68c06edb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd694036000 tx=0x7fd694008030 comp rx=0 tx=0).stop 2024-04-15T15:15:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 msgr2=0x7fd6ac150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac150c80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c4d0 tx=0x7fd6a400c990 comp rx=0 tx=0).stop 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 shutdown_connections 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd68c06c8f0 0x7fd68c06edb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6ac0d0bd0 0x7fd6ac150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd6ac072d80 0x7fd6ac15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 --2- 172.21.15.46:0/2917642867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac071d10 0x7fd6ac150c80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.184+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 >> 172.21.15.46:0/2917642867 conn(0x7fd6ac06be00 msgr2=0x7fd6ac0736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.185+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 shutdown_connections 2024-04-15T15:15:52.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.185+0000 7fd6b17f2640 1 -- 172.21.15.46:0/2917642867 wait complete. 2024-04-15T15:15:52.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.301+0000 7fe84e474640 1 Processor -- start 2024-04-15T15:15:52.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.301+0000 7fe84e474640 1 -- start start 2024-04-15T15:15:52.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe84e474640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8480d3240 0x7fe8480d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe84e474640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe8480b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe84e474640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8480d2480 con 0x7fe84815e790 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe84e474640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe8480d2db0 con 0x7fe8480d3240 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe84e474640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe8480d2760 con 0x7fe8480d1440 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8480d3240 0x7fe8480d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe8480d3240 0x7fe8480d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60368/0 (socket says 172.21.15.46:60368) 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 -- 172.21.15.46:0/290943794 learned_addr learned my addr 172.21.15.46:0/290943794 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe84ca11640 1 --2- 172.21.15.46:0/290943794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe8480b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 -- 172.21.15.46:0/290943794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 msgr2=0x7fe8480b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 --2- 172.21.15.46:0/290943794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe8480b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 -- 172.21.15.46:0/290943794 >> v1:172.21.15.78:6789/0 conn(0x7fe84815e790 legacy=0x7fe8480cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.302+0000 7fe83ffff640 1 -- 172.21.15.46:0/290943794 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8480d2a40 con 0x7fe8480d3240 2024-04-15T15:15:52.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.303+0000 7fe83ffff640 1 --2- 172.21.15.46:0/290943794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480d0f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe830009870 tx=0x7fe83002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f048b078a760c41 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.303+0000 7fe8477fe640 1 -- 172.21.15.46:0/290943794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830034070 con 0x7fe8480d3240 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.303+0000 7fe8477fe640 1 -- 172.21.15.46:0/290943794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe830030840 con 0x7fe8480d3240 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.303+0000 7fe8477fe640 1 -- 172.21.15.46:0/290943794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830030c10 con 0x7fe8480d3240 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 -- 172.21.15.46:0/290943794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 msgr2=0x7fe8480d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 --2- 172.21.15.46:0/290943794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480d0f00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe830009870 tx=0x7fe83002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 -- 172.21.15.46:0/290943794 shutdown_connections 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 --2- 172.21.15.46:0/290943794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe8480b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 --2- 172.21.15.46:0/290943794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480d0f00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 -- 172.21.15.46:0/290943794 >> 172.21.15.46:0/290943794 conn(0x7fe84808bf60 msgr2=0x7fe8480d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 -- 172.21.15.46:0/290943794 shutdown_connections 2024-04-15T15:15:52.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.304+0000 7fe84e474640 1 -- 172.21.15.46:0/290943794 wait complete. 2024-04-15T15:15:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.305+0000 7fe84e474640 1 Processor -- start 2024-04-15T15:15:52.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.305+0000 7fe84e474640 1 -- start start 2024-04-15T15:15:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84e474640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe848147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84e474640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84e474640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe84815e790 0x7fe8480cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84e474640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8480d5180 con 0x7fe8480d3240 2024-04-15T15:15:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84e474640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe8480d0430 con 0x7fe84815e790 2024-04-15T15:15:52.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84e474640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe8480d5300 con 0x7fe8480d1440 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480cd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60378/0 (socket says 172.21.15.46:60378) 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 -- 172.21.15.46:0/919098882 learned_addr learned my addr 172.21.15.46:0/919098882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84ca11640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe84815e790 0x7fe8480cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 -- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 msgr2=0x7fe848147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe847fff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe848147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe848147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 -- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe84815e790 msgr2=0x7fe8480cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe84815e790 0x7fe8480cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe83ffff640 1 -- 172.21.15.46:0/919098882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe848146b60 con 0x7fe8480d3240 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe84ca11640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe84815e790 0x7fe8480cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.306+0000 7fe847fff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe848147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.307+0000 7fe83ffff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480cd780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe830000c00 tx=0x7fe830004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.307+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830048070 con 0x7fe8480d3240 2024-04-15T15:15:52.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.307+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe830030640 con 0x7fe8480d3240 2024-04-15T15:15:52.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.307+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe848146e40 con 0x7fe8480d3240 2024-04-15T15:15:52.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.307+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830034040 con 0x7fe8480d3240 2024-04-15T15:15:52.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.307+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe848147400 con 0x7fe8480d3240 2024-04-15T15:15:52.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.308+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84808e100 con 0x7fe8480d3240 2024-04-15T15:15:52.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.308+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe83004b070 con 0x7fe8480d3240 2024-04-15T15:15:52.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.309+0000 7fe8457fa640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe82806c8f0 0x7fe82806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.309+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe8300add30 con 0x7fe8480d3240 2024-04-15T15:15:52.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.309+0000 7fe847fff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe82806c8f0 0x7fe82806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.309+0000 7fe847fff640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe82806c8f0 0x7fe82806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8380015e0 tx=0x7fe838009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.313+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe84808e100 con 0x7fe8480d3240 2024-04-15T15:15:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.510+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe8480d0af0 con 0x7fe8480d3240 2024-04-15T15:15:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.510+0000 7fe8457fa640 1 -- 172.21.15.46:0/919098882 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7fe8480d0af0 con 0x7fe8480d3240 2024-04-15T15:15:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe82806c8f0 msgr2=0x7fe82806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe82806c8f0 0x7fe82806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe8380015e0 tx=0x7fe838009040 comp rx=0 tx=0).stop 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 msgr2=0x7fe8480cd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480cd780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe830000c00 tx=0x7fe830004730 comp rx=0 tx=0).stop 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 shutdown_connections 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe82806c8f0 0x7fe82806edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe84815e790 0x7fe8480cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8480d3240 0x7fe8480cd780 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 --2- 172.21.15.46:0/919098882 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8480d1440 0x7fe848147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 >> 172.21.15.46:0/919098882 conn(0x7fe84808bf60 msgr2=0x7fe8480b8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 shutdown_connections 2024-04-15T15:15:52.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:52.513+0000 7fe84e474640 1 -- 172.21.15.46:0/919098882 wait complete. 2024-04-15T15:15:52.526 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-04-15T15:15:52.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:52.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:52.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[44463]: fuse finished with error 0 and tester_r 0 2024-04-15T15:15:52.556 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:58.657 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:15:58.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:58.658 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-15T15:15:58.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.843+0000 7f2bbc47b640 1 Processor -- start 2024-04-15T15:15:58.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.843+0000 7f2bbc47b640 1 -- start start 2024-04-15T15:15:58.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bbc47b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2bb4070940 0x7f2bb409b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bbc47b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2bb4160310 0x7f2bb4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bbc47b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb40d05e0 con 0x7f2bb40d0da0 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bbc47b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2bb4071260 con 0x7f2bb4070940 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bbc47b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2bb415f720 con 0x7f2bb4160310 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2bb4070940 0x7f2bb409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2bb4070940 0x7f2bb409b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60382/0 (socket says 172.21.15.46:60382) 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 -- 172.21.15.46:0/1710860722 learned_addr learned my addr 172.21.15.46:0/1710860722 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bb9a16640 1 --2- 172.21.15.46:0/1710860722 >> v2:172.21.15.78:3300/0 conn(0x7f2bb4160310 0x7f2bb4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 -- 172.21.15.46:0/1710860722 >> v2:172.21.15.78:3300/0 conn(0x7f2bb4160310 msgr2=0x7f2bb4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 --2- 172.21.15.46:0/1710860722 >> v2:172.21.15.78:3300/0 conn(0x7f2bb4160310 0x7f2bb4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 -- 172.21.15.46:0/1710860722 >> v1:172.21.15.78:6789/0 conn(0x7f2bb40d0da0 legacy=0x7f2bb40d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bba217640 1 -- 172.21.15.46:0/1710860722 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb415fa00 con 0x7f2bb4070940 2024-04-15T15:15:58.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.844+0000 7f2bb9a16640 1 --2- 172.21.15.46:0/1710860722 >> v2:172.21.15.78:3300/0 conn(0x7f2bb4160310 0x7f2bb4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.845+0000 7f2bba217640 1 --2- 172.21.15.46:0/1710860722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4070940 0x7f2bb409b880 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40734f0 tx=0x7f2bac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b9350d4bb25b23 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.845+0000 7f2bb9215640 1 -- 172.21.15.46:0/1710860722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac035020 con 0x7f2bb4070940 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.845+0000 7f2bb9215640 1 -- 172.21.15.46:0/1710860722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bac031870 con 0x7f2bb4070940 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.845+0000 7f2bb9215640 1 -- 172.21.15.46:0/1710860722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac031c40 con 0x7f2bb4070940 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 -- 172.21.15.46:0/1710860722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4070940 msgr2=0x7f2bb409b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/1710860722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4070940 0x7f2bb409b880 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2bb40734f0 tx=0x7f2bac02b910 comp rx=0 tx=0).stop 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 -- 172.21.15.46:0/1710860722 shutdown_connections 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/1710860722 >> v2:172.21.15.78:3300/0 conn(0x7f2bb4160310 0x7f2bb4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/1710860722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4070940 0x7f2bb409b880 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 -- 172.21.15.46:0/1710860722 >> 172.21.15.46:0/1710860722 conn(0x7f2bb40b3f40 msgr2=0x7f2bb40b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 -- 172.21.15.46:0/1710860722 shutdown_connections 2024-04-15T15:15:58.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.846+0000 7f2bbc47b640 1 -- 172.21.15.46:0/1710860722 wait complete. 2024-04-15T15:15:58.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.847+0000 7f2bbc47b640 1 Processor -- start 2024-04-15T15:15:58.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.847+0000 7f2bbc47b640 1 -- start start 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbc47b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bb4070940 0x7f2bb4154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbc47b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40d0da0 0x7f2bb406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbc47b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 0x7f2bb406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbc47b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb40d18e0 con 0x7f2bb4160310 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbc47b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb40d0880 con 0x7f2bb40d0da0 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbc47b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb4160770 con 0x7f2bb4070940 2024-04-15T15:15:58.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbaa18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 0x7f2bb406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbaa18640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 0x7f2bb406e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60386/0 (socket says 172.21.15.46:60386) 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bbaa18640 1 -- 172.21.15.46:0/3267576652 learned_addr learned my addr 172.21.15.46:0/3267576652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bba217640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bb4070940 0x7f2bb4154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.848+0000 7f2bb9a16640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40d0da0 0x7f2bb406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbaa18640 1 -- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bb4070940 msgr2=0x7f2bb4154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbaa18640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bb4070940 0x7f2bb4154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbaa18640 1 -- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40d0da0 msgr2=0x7f2bb406ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbaa18640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40d0da0 0x7f2bb406ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbaa18640 1 -- 172.21.15.46:0/3267576652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb41f8110 con 0x7f2bb4160310 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bb9a16640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40d0da0 0x7f2bb406ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbaa18640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 0x7f2bb406e2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400b9a0 tx=0x7f2ba400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4019070 con 0x7f2bb4160310 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ba400cba0 con 0x7f2bb4160310 2024-04-15T15:15:58.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb41f83f0 con 0x7f2bb4160310 2024-04-15T15:15:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4014440 con 0x7f2bb4160310 2024-04-15T15:15:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.849+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb4154f50 con 0x7f2bb4160310 2024-04-15T15:15:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.850+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb40689e0 con 0x7f2bb4160310 2024-04-15T15:15:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.851+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ba401c070 con 0x7f2bb4160310 2024-04-15T15:15:58.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.851+0000 7f2bb2ffd640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2ba006c8f0 0x7f2ba006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.851+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2ba408bba0 con 0x7f2bb4160310 2024-04-15T15:15:58.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.851+0000 7f2bba217640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2ba006c8f0 0x7f2ba006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.852+0000 7f2bba217640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2ba006c8f0 0x7f2ba006edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2bac036000 tx=0x7f2bac03c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:58.856+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ba4059c90 con 0x7f2bb4160310 2024-04-15T15:15:59.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.041+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bb409b970 con 0x7f2bb4160310 2024-04-15T15:15:59.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.042+0000 7f2bb2ffd640 1 -- 172.21.15.46:0/3267576652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f2ba408b020 con 0x7f2bb4160310 2024-04-15T15:15:59.044 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:59.044 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2ba006c8f0 msgr2=0x7f2ba006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2ba006c8f0 0x7f2ba006edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2bac036000 tx=0x7f2bac03c040 comp rx=0 tx=0).stop 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 msgr2=0x7f2bb406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 0x7f2bb406e2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2ba400b9a0 tx=0x7f2ba400be60 comp rx=0 tx=0).stop 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 shutdown_connections 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2ba006c8f0 0x7f2ba006edb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4160310 0x7f2bb406e2f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40d0da0 0x7f2bb406ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 --2- 172.21.15.46:0/3267576652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bb4070940 0x7f2bb4154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 >> 172.21.15.46:0/3267576652 conn(0x7f2bb40b3f40 msgr2=0x7f2bb40b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 shutdown_connections 2024-04-15T15:15:59.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.044+0000 7f2bbc47b640 1 -- 172.21.15.46:0/3267576652 wait complete. 2024-04-15T15:15:59.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:59.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.160+0000 7f61d1550640 1 Processor -- start 2024-04-15T15:15:59.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.160+0000 7f61d1550640 1 -- start start 2024-04-15T15:15:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.161+0000 7f61d1550640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f61cc15c8d0 0x7f61cc0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.161+0000 7f61d1550640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61cc0d0350 con 0x7f61cc09b650 2024-04-15T15:15:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.161+0000 7f61d1550640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61cc0ce100 con 0x7f61cc09b0e0 2024-04-15T15:15:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.161+0000 7f61d1550640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f61cc0b5b00 con 0x7f61cc15c8d0 2024-04-15T15:15:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.161+0000 7f61ca7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f61cc09b650 0x7f61cc09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41542/0 (socket says 172.21.15.46:41542) 2024-04-15T15:15:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.161+0000 7f61ca7fc640 1 -- 172.21.15.46:0/4132622130 learned_addr learned my addr 172.21.15.46:0/4132622130 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61caffd640 1 --2- 172.21.15.46:0/4132622130 >> v2:172.21.15.78:3300/0 conn(0x7f61cc15c8d0 0x7f61cc0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2548620713 0 0) 0x7f61cc0d0350 con 0x7f61cc09b650 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61b8003520 con 0x7f61cc09b650 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212782805 0 0) 0x7f61cc0ce100 con 0x7f61cc09b0e0 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61cc0d0350 con 0x7f61cc09b0e0 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1148920494 0 0) 0x7f61b8003520 con 0x7f61cc09b650 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 >> v2:172.21.15.78:3300/0 conn(0x7f61cc15c8d0 msgr2=0x7f61cc0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 --2- 172.21.15.46:0/4132622130 >> v2:172.21.15.78:3300/0 conn(0x7f61cc15c8d0 0x7f61cc0cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 >> v1:172.21.15.78:6789/0 conn(0x7f61cc09b0e0 legacy=0x7f61cc0b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc0b54b0 con 0x7f61cc09b650 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f61b4001670 con 0x7f61cc09b650 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61caffd640 1 --2- 172.21.15.46:0/4132622130 >> v2:172.21.15.78:3300/0 conn(0x7f61cc15c8d0 0x7f61cc0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 >> v1:172.21.15.46:6789/0 conn(0x7f61cc09b650 legacy=0x7f61cc09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b8003b40 0x7f61b8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61b80051c0 0x7f61b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc0b5b00 con 0x7f61b8004800 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f61b800c0f0 con 0x7f61b8003b40 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f61b800c2f0 con 0x7f61b80051c0 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61caffd640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b8003b40 0x7f61b8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61cb7fe640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61b80051c0 0x7f61b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61ca7fc640 1 -- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61b80051c0 msgr2=0x7f61b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61b80051c0 0x7f61b800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61ca7fc640 1 -- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b8003b40 msgr2=0x7f61b8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b8003b40 0x7f61b8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61ca7fc640 1 -- 172.21.15.46:0/4132622130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b8003520 con 0x7f61b8004800 2024-04-15T15:15:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.162+0000 7f61cb7fe640 1 -- 172.21.15.46:0/4132622130 reap_dead start 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.163+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61b8004c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c190 tx=0x7f61b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f84b3ea04a1305 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.163+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4007ce0 con 0x7f61b8004800 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.163+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b4030070 con 0x7f61b8004800 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.163+0000 7f61c9ffb640 1 -- 172.21.15.46:0/4132622130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4005d90 con 0x7f61b8004800 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 -- 172.21.15.46:0/4132622130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 msgr2=0x7f61b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61b8004c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c190 tx=0x7f61b8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 -- 172.21.15.46:0/4132622130 shutdown_connections 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 --2- 172.21.15.46:0/4132622130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61b8004c80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 -- 172.21.15.46:0/4132622130 >> 172.21.15.46:0/4132622130 conn(0x7f61cc0b3fb0 msgr2=0x7f61cc0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 -- 172.21.15.46:0/4132622130 shutdown_connections 2024-04-15T15:15:59.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.164+0000 7f61d1550640 1 -- 172.21.15.46:0/4132622130 wait complete. 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.165+0000 7f61d1550640 1 Processor -- start 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.165+0000 7f61d1550640 1 -- start start 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61d1550640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61cc1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61d1550640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61cc1e6c10 0x7f61cc156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61d1550640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61cc157000 0x7f61cc1f7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61d1550640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc0cdbf0 con 0x7f61b8004800 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61d1550640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc0cda70 con 0x7f61cc1e6c10 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61d1550640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f61cc0cdd70 con 0x7f61cc157000 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61cc1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61cc1478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60404/0 (socket says 172.21.15.46:60404) 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 -- 172.21.15.46:0/4175177777 learned_addr learned my addr 172.21.15.46:0/4175177777 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61cc1e6c10 0x7f61cc156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61cb7fe640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61cc157000 0x7f61cc1f7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 -- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61cc157000 msgr2=0x7f61cc1f7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61cc157000 0x7f61cc1f7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 -- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61cc1e6c10 msgr2=0x7f61cc156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61cc1e6c10 0x7f61cc156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.166+0000 7f61caffd640 1 -- 172.21.15.46:0/4175177777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc147df0 con 0x7f61b8004800 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.167+0000 7f61caffd640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61cc1478b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f61c4011c00 tx=0x7f61c400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.167+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c400fcc0 con 0x7f61b8004800 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.167+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c400fe60 con 0x7f61b8004800 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.167+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61cc155a20 con 0x7f61b8004800 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.167+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4004270 con 0x7f61b8004800 2024-04-15T15:15:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.167+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61cc155fe0 con 0x7f61b8004800 2024-04-15T15:15:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.168+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61cc0690f0 con 0x7f61b8004800 2024-04-15T15:15:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.168+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f61c4012040 con 0x7f61b8004800 2024-04-15T15:15:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.169+0000 7f61c37fe640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61ac06c8f0 0x7f61ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.169+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f61c408c290 con 0x7f61b8004800 2024-04-15T15:15:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.169+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61ac06c8f0 0x7f61ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.169+0000 7f61ca7fc640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61ac06c8f0 0x7f61ac06edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f61b4034000 tx=0x7f61b4039040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.173+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f61cc147df0 con 0x7f61b8004800 2024-04-15T15:15:59.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.358+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61cc0cecb0 con 0x7f61b8004800 2024-04-15T15:15:59.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.359+0000 7f61c37fe640 1 -- 172.21.15.46:0/4175177777 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f61c4019070 con 0x7f61b8004800 2024-04-15T15:15:59.361 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:15:59.361 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61ac06c8f0 msgr2=0x7f61ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61ac06c8f0 0x7f61ac06edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f61b4034000 tx=0x7f61b4039040 comp rx=0 tx=0).stop 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 msgr2=0x7f61cc1478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61cc1478b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f61c4011c00 tx=0x7f61c400f610 comp rx=0 tx=0).stop 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 shutdown_connections 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f61ac06c8f0 0x7f61ac06edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f61cc157000 0x7f61cc1f7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61cc1e6c10 0x7f61cc156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 --2- 172.21.15.46:0/4175177777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61b8004800 0x7f61cc1478b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 >> 172.21.15.46:0/4175177777 conn(0x7f61cc0b3fb0 msgr2=0x7f61cc1e3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 shutdown_connections 2024-04-15T15:15:59.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.361+0000 7f61d1550640 1 -- 172.21.15.46:0/4175177777 wait complete. 2024-04-15T15:15:59.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.479+0000 7fa6f9959640 1 Processor -- start 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.479+0000 7fa6f9959640 1 -- start start 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f9959640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f40b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f9959640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa6f415c250 0x7fa6f40b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f9959640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f40b1290 con 0x7fa6f40b2330 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f9959640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f40b1070 con 0x7fa6f415c250 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f9959640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f40afdb0 con 0x7fa6f415b760 2024-04-15T15:15:59.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa6f40b2330 0x7fa6f415bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:41554/0 (socket says 172.21.15.46:41554) 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.480+0000 7fa6f27fc640 1 -- 172.21.15.46:0/2779894669 learned_addr learned my addr 172.21.15.46:0/2779894669 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f2ffd640 1 --2- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f40b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f37fe640 1 --2- 172.21.15.46:0/2779894669 >> v2:172.21.15.78:3300/0 conn(0x7fa6f415c250 0x7fa6f40b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f1ffb640 1 -- 172.21.15.46:0/2779894669 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807509482 0 0) 0x7fa6f40b1290 con 0x7fa6f40b2330 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f1ffb640 1 -- 172.21.15.46:0/2779894669 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6e0003520 con 0x7fa6f40b2330 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f37fe640 1 -- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 msgr2=0x7fa6f40b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f37fe640 1 --2- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f40b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f37fe640 1 -- 172.21.15.46:0/2779894669 >> v1:172.21.15.46:6789/0 conn(0x7fa6f40b2330 legacy=0x7fa6f415bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f1ffb640 1 -- 172.21.15.46:0/2779894669 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2177210572 0 0) 0x7fa6e0003520 con 0x7fa6f40b2330 2024-04-15T15:15:59.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.481+0000 7fa6f37fe640 1 -- 172.21.15.46:0/2779894669 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f40b1570 con 0x7fa6f415c250 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f37fe640 1 --2- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f415c250 0x7fa6f40b27b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40097a0 tx=0x7fa6e402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7554b00840ba87a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f1ffb640 1 -- 172.21.15.46:0/2779894669 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4037070 con 0x7fa6f415c250 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f1ffb640 1 -- 172.21.15.46:0/2779894669 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e4030510 con 0x7fa6f415c250 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f1ffb640 1 -- 172.21.15.46:0/2779894669 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc001670 con 0x7fa6f415c250 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 -- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f415c250 msgr2=0x7fa6f40b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 --2- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f415c250 0x7fa6f40b27b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40097a0 tx=0x7fa6e402b790 comp rx=0 tx=0).stop 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 -- 172.21.15.46:0/2779894669 shutdown_connections 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 --2- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f415c250 0x7fa6f40b27b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 --2- 172.21.15.46:0/2779894669 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f40b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 -- 172.21.15.46:0/2779894669 >> 172.21.15.46:0/2779894669 conn(0x7fa6f4073260 msgr2=0x7fa6f409a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.482+0000 7fa6f9959640 1 -- 172.21.15.46:0/2779894669 shutdown_connections 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.483+0000 7fa6f9959640 1 -- 172.21.15.46:0/2779894669 wait complete. 2024-04-15T15:15:59.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.484+0000 7fa6f9959640 1 Processor -- start 2024-04-15T15:15:59.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.484+0000 7fa6f9959640 1 -- start start 2024-04-15T15:15:59.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.484+0000 7fa6f9959640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f40b2330 0x7fa6f40d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.484+0000 7fa6f9959640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f414ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f9959640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 0x7fa6f41569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f9959640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f40b1e10 con 0x7fa6f415c250 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f9959640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f40b1b30 con 0x7fa6f40b2330 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f9959640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f406b9e0 con 0x7fa6f415b760 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 0x7fa6f41569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 0x7fa6f41569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60412/0 (socket says 172.21.15.46:60412) 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f27fc640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f414ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f2ffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f40b2330 0x7fa6f40d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 -- 172.21.15.46:0/1559806939 learned_addr learned my addr 172.21.15.46:0/1559806939 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 -- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 msgr2=0x7fa6f414ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f414ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 -- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f40b2330 msgr2=0x7fa6f40d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f40b2330 0x7fa6f40d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 -- 172.21.15.46:0/1559806939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f41f4b00 con 0x7fa6f415c250 2024-04-15T15:15:59.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f37fe640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 0x7fa6f41569b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40097a0 tx=0x7fa6e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4041020 con 0x7fa6f415c250 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e403c870 con 0x7fa6f415c250 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f41f4de0 con 0x7fa6f415c250 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4037040 con 0x7fa6f415c250 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.485+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f41f53a0 con 0x7fa6f415c250 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.486+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6f40690f0 con 0x7fa6f415c250 2024-04-15T15:15:59.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.487+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa6e4044020 con 0x7fa6f415c250 2024-04-15T15:15:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.487+0000 7fa6ebfff640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa6d406c920 0x7fa6d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.487+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa6e40ad550 con 0x7fa6f415c250 2024-04-15T15:15:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.488+0000 7fa6f2ffd640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa6d406c920 0x7fa6d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.488+0000 7fa6f2ffd640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa6d406c920 0x7fa6d406ede0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0113e0 tx=0x7fa6ec00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.492+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6e404b070 con 0x7fa6f415c250 2024-04-15T15:15:59.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:15:59.715+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fa6f41552f0 con 0x7fa6f415c250 2024-04-15T15:16:00.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.628+0000 7fa6ebfff640 1 -- 172.21.15.46:0/1559806939 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7fa6e407af50 con 0x7fa6f415c250 2024-04-15T15:16:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa6d406c920 msgr2=0x7fa6d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa6d406c920 0x7fa6d406ede0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0113e0 tx=0x7fa6ec00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 msgr2=0x7fa6f41569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 0x7fa6f41569b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40097a0 tx=0x7fa6e40046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 shutdown_connections 2024-04-15T15:16:00.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa6d406c920 0x7fa6d406ede0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6f415c250 0x7fa6f41569b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa6f415b760 0x7fa6f414ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 --2- 172.21.15.46:0/1559806939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6f40b2330 0x7fa6f40d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 >> 172.21.15.46:0/1559806939 conn(0x7fa6f4073260 msgr2=0x7fa6f4159d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 shutdown_connections 2024-04-15T15:16:00.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.631+0000 7fa6f9959640 1 -- 172.21.15.46:0/1559806939 wait complete. 2024-04-15T15:16:00.646 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-15T15:16:00.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:16:00.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.759+0000 7f7877ac5640 1 Processor -- start 2024-04-15T15:16:00.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.759+0000 7f7877ac5640 1 -- start start 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7877ac5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f787015e790 0x7f78700d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7877ac5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78700d2480 con 0x7f78700d1610 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7877ac5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78700d2760 con 0x7f78700cfd20 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7877ac5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f78700d2db0 con 0x7f787015e790 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7876062640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f78700d1610 0x7f78700b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55558/0 (socket says 172.21.15.46:55558) 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7876062640 1 -- 172.21.15.46:0/2021191343 learned_addr learned my addr 172.21.15.46:0/2021191343 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7875861640 1 --2- 172.21.15.46:0/2021191343 >> v2:172.21.15.78:3300/0 conn(0x7f787015e790 0x7f78700d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161036237 0 0) 0x7f78700d2480 con 0x7f78700d1610 2024-04-15T15:16:00.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f785c003520 con 0x7f78700d1610 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1208626990 0 0) 0x7f785c003520 con 0x7f78700d1610 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 >> v2:172.21.15.78:3300/0 conn(0x7f787015e790 msgr2=0x7f78700d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 --2- 172.21.15.46:0/2021191343 >> v2:172.21.15.78:3300/0 conn(0x7f787015e790 0x7f78700d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 >> v1:172.21.15.78:6789/0 conn(0x7f78700cfd20 legacy=0x7f78700d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78700d2a40 con 0x7f78700d1610 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f78680019f0 con 0x7f78700d1610 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f7875861640 1 --2- 172.21.15.46:0/2021191343 >> v2:172.21.15.78:3300/0 conn(0x7f787015e790 0x7f78700d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 >> v1:172.21.15.46:6789/0 conn(0x7f78700d1610 legacy=0x7f78700b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f785c0039b0 0x7f785c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c004800 0x7f785c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f785c0051c0 0x7f785c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78700d2db0 con 0x7f785c004800 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f785c00c090 con 0x7f785c0039b0 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.760+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f785c00c290 con 0x7f785c0051c0 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7876062640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f785c0051c0 0x7f785c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875861640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f785c0039b0 0x7f785c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875060640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c004800 0x7f785c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875861640 1 -- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f785c0051c0 msgr2=0x7f785c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875861640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f785c0051c0 0x7f785c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875861640 1 -- 172.21.15.46:0/2021191343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c004800 msgr2=0x7f785c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875861640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c004800 0x7f785c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875861640 1 -- 172.21.15.46:0/2021191343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c003520 con 0x7f785c0039b0 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7876062640 1 -- 172.21.15.46:0/2021191343 reap_dead start 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.761+0000 7f7875060640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c004800 0x7f785c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:00.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.762+0000 7f7875861640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f785c0039b0 0x7f785c003df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f786c009cc0 tx=0x7f785c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a978eea05f13d6ec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.762+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c039070 con 0x7f785c0039b0 2024-04-15T15:16:00.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.762+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786c02c8d0 con 0x7f785c0039b0 2024-04-15T15:16:00.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.762+0000 7f787485f640 1 -- 172.21.15.46:0/2021191343 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c009860 con 0x7f785c0039b0 2024-04-15T15:16:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.762+0000 7f7877ac5640 1 -- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f785c0039b0 msgr2=0x7f785c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.762+0000 7f7877ac5640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f785c0039b0 0x7f785c003df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f786c009cc0 tx=0x7f785c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.763+0000 7f7877ac5640 1 -- 172.21.15.46:0/2021191343 shutdown_connections 2024-04-15T15:16:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.763+0000 7f7877ac5640 1 --2- 172.21.15.46:0/2021191343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f785c0039b0 0x7f785c003df0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.763+0000 7f7877ac5640 1 -- 172.21.15.46:0/2021191343 >> 172.21.15.46:0/2021191343 conn(0x7f787008bf60 msgr2=0x7f78700d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.763+0000 7f7877ac5640 1 -- 172.21.15.46:0/2021191343 shutdown_connections 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.763+0000 7f7877ac5640 1 -- 172.21.15.46:0/2021191343 wait complete. 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.764+0000 7f7877ac5640 1 Processor -- start 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.764+0000 7f7877ac5640 1 -- start start 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.764+0000 7f7877ac5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 0x7f787015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.764+0000 7f7877ac5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f78700cc3d0 0x7f78700cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.764+0000 7f7877ac5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c9cf0 0x7f78700ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7877ac5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f787015ecf0 con 0x7f785c0039b0 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7877ac5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f787015eb70 con 0x7f78700c9cf0 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7877ac5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f78700d0160 con 0x7f78700cc3d0 2024-04-15T15:16:00.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 0x7f787015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 0x7f787015dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41982/0 (socket says 172.21.15.46:41982) 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 -- 172.21.15.46:0/3460126770 learned_addr learned my addr 172.21.15.46:0/3460126770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7876062640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c9cf0 0x7f78700ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875060640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f78700cc3d0 0x7f78700cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 -- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f78700cc3d0 msgr2=0x7f78700cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f78700cc3d0 0x7f78700cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 -- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c9cf0 msgr2=0x7f78700ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c9cf0 0x7f78700ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 -- 172.21.15.46:0/3460126770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78701f45d0 con 0x7f785c0039b0 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7875861640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 0x7f787015dc20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f786c009cc0 tx=0x7f786c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c039050 con 0x7f785c0039b0 2024-04-15T15:16:00.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786c03a040 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78700cdfc0 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c009860 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.765+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78700ce580 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.766+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78700689e0 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.767+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f786c047030 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.767+0000 7f78667fc640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785406c8f0 0x7f785406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.767+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f786c0ad1e0 con 0x7f785c0039b0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.768+0000 7f7875060640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785406c8f0 0x7f785406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.768+0000 7f7875060640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785406c8f0 0x7f785406edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78700cc7b0 tx=0x7f7860009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.772+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f786c07ac00 con 0x7f785c0039b0 2024-04-15T15:16:00.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:00.994+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f78700d1ce0 con 0x7f785c0039b0 2024-04-15T15:16:01.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.629+0000 7f78667fc640 1 -- 172.21.15.46:0/3460126770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f786c07ef00 con 0x7f785c0039b0 2024-04-15T15:16:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785406c8f0 msgr2=0x7f785406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785406c8f0 0x7f785406edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f78700cc7b0 tx=0x7f7860009040 comp rx=0 tx=0).stop 2024-04-15T15:16:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 msgr2=0x7f787015dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 0x7f787015dc20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f786c009cc0 tx=0x7f786c004730 comp rx=0 tx=0).stop 2024-04-15T15:16:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 shutdown_connections 2024-04-15T15:16:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785406c8f0 0x7f785406edb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c9cf0 0x7f78700ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f78700cc3d0 0x7f78700cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 --2- 172.21.15.46:0/3460126770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c0039b0 0x7f787015dc20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 >> 172.21.15.46:0/3460126770 conn(0x7f787008bf60 msgr2=0x7f78700b9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 shutdown_connections 2024-04-15T15:16:01.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.632+0000 7f7877ac5640 1 -- 172.21.15.46:0/3460126770 wait complete. 2024-04-15T15:16:01.648 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:16:01.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.757+0000 7f2fb2093640 1 Processor -- start 2024-04-15T15:16:01.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.757+0000 7f2fb2093640 1 -- start start 2024-04-15T15:16:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fb2093640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fac15e700 0x7f2fac0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fb2093640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac0d2360 0x7f2fac0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fb2093640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fac0b47a0 con 0x7f2fac15eae0 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fb2093640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2fac0b53b0 con 0x7f2fac15e700 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fb2093640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2fac0b4a80 con 0x7f2fac0d2360 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fab7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fac15e700 0x7f2fac0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fab7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2fac15e700 0x7f2fac0b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41984/0 (socket says 172.21.15.46:41984) 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.758+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3027500094 learned_addr learned my addr 172.21.15.46:0/3027500094 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.759+0000 7f2faaffd640 1 --2- 172.21.15.46:0/3027500094 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac0d2360 0x7f2fac0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.759+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3027500094 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac0d2360 msgr2=0x7f2fac0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.759+0000 7f2fab7fe640 1 --2- 172.21.15.46:0/3027500094 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac0d2360 0x7f2fac0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.759+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3027500094 >> v1:172.21.15.78:6789/0 conn(0x7f2fac15eae0 legacy=0x7f2fac0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.759+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3027500094 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac0b4d60 con 0x7f2fac15e700 2024-04-15T15:16:01.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fab7fe640 1 --2- 172.21.15.46:0/3027500094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac15e700 0x7f2fac0b5840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009ea0 tx=0x7f2f9c02bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f09b1deced4e5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2faa7fc640 1 -- 172.21.15.46:0/3027500094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c02cb40 con 0x7f2fac15e700 2024-04-15T15:16:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2faa7fc640 1 -- 172.21.15.46:0/3027500094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9c02cce0 con 0x7f2fac15e700 2024-04-15T15:16:01.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2faa7fc640 1 -- 172.21.15.46:0/3027500094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c034610 con 0x7f2fac15e700 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 -- 172.21.15.46:0/3027500094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac15e700 msgr2=0x7f2fac0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3027500094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac15e700 0x7f2fac0b5840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009ea0 tx=0x7f2f9c02bfa0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 -- 172.21.15.46:0/3027500094 shutdown_connections 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3027500094 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac0d2360 0x7f2fac0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3027500094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac15e700 0x7f2fac0b5840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 -- 172.21.15.46:0/3027500094 >> 172.21.15.46:0/3027500094 conn(0x7f2fac08bf60 msgr2=0x7f2fac0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 -- 172.21.15.46:0/3027500094 shutdown_connections 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.760+0000 7f2fb2093640 1 -- 172.21.15.46:0/3027500094 wait complete. 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 Processor -- start 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 -- start start 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 0x7f2fac1519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac15e700 0x7f2fac0b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fac15eae0 0x7f2fac0b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac0b5d80 con 0x7f2fac0d2360 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac0b4450 con 0x7f2fac15eae0 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.762+0000 7f2fb2093640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2fac0d1d40 con 0x7f2fac15e700 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 0x7f2fac1519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 0x7f2fac1519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41988/0 (socket says 172.21.15.46:41988) 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3751999515 learned_addr learned my addr 172.21.15.46:0/3751999515 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2faaffd640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac15e700 0x7f2fac0b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fabfff640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fac15eae0 0x7f2fac0b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac15e700 msgr2=0x7f2fac0b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac15e700 0x7f2fac0b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fac15eae0 msgr2=0x7f2fac0b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fac15eae0 0x7f2fac0b8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 -- 172.21.15.46:0/3751999515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac0b9b20 con 0x7f2fac0d2360 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2faaffd640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac15e700 0x7f2fac0b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fab7fe640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 0x7f2fac1519f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c036040 tx=0x7f2f9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c043020 con 0x7f2fac0d2360 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9c03e450 con 0x7f2fac0d2360 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fac0b9e00 con 0x7f2fac0d2360 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c009510 con 0x7f2fac0d2360 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.763+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fac15ad70 con 0x7f2fac0d2360 2024-04-15T15:16:01.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.764+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fac0689e0 con 0x7f2fac0d2360 2024-04-15T15:16:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.765+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f9c046020 con 0x7f2fac0d2360 2024-04-15T15:16:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.765+0000 7f2f93fff640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.765+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2f9c0ae710 con 0x7f2fac0d2360 2024-04-15T15:16:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.766+0000 7f2faaffd640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.766+0000 7f2faaffd640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f9806c8f0 0x7f2f9806edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2fac15ab60 tx=0x7f2fa0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.770+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f9c04d030 con 0x7f2fac0d2360 2024-04-15T15:16:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.991+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fac0ba500 con 0x7f2fac0d2360 2024-04-15T15:16:01.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.991+0000 7f2f93fff640 1 -- 172.21.15.46:0/3751999515 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f2f9c07c800 con 0x7f2fac0d2360 2024-04-15T15:16:01.993 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:01.993 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:01.994 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f9806c8f0 msgr2=0x7f2f9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f9806c8f0 0x7f2f9806edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2fac15ab60 tx=0x7f2fa0008040 comp rx=0 tx=0).stop 2024-04-15T15:16:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 msgr2=0x7f2fac1519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 0x7f2fac1519f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c036040 tx=0x7f2f9c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:01.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 shutdown_connections 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f9806c8f0 0x7f2f9806edb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2fac15eae0 0x7f2fac0b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2fac15e700 0x7f2fac0b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 --2- 172.21.15.46:0/3751999515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fac0d2360 0x7f2fac1519f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 >> 172.21.15.46:0/3751999515 conn(0x7f2fac08bf60 msgr2=0x7f2fac0d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 shutdown_connections 2024-04-15T15:16:01.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:01.993+0000 7f2fb2093640 1 -- 172.21.15.46:0/3751999515 wait complete. 2024-04-15T15:16:02.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.110+0000 7fb9a99af640 1 Processor -- start 2024-04-15T15:16:02.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.110+0000 7fb9a99af640 1 -- start start 2024-04-15T15:16:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a99af640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb9a41602d0 0x7fb9a40cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a99af640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a409afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a99af640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a415dbd0 con 0x7fb9a41606b0 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a99af640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a40cd6a0 con 0x7fb9a41602d0 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a99af640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a40cd980 con 0x7fb9a409ab40 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a2ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb9a41602d0 0x7fb9a40cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a409afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a409afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42560/0 (socket says 172.21.15.46:42560) 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb99bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb9a41606b0 0x7fb9a40d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47050/0 (socket says 172.21.15.46:47050) 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 -- 172.21.15.46:0/1054384080 learned_addr learned my addr 172.21.15.46:0/1054384080 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a27fc640 1 -- 172.21.15.46:0/1054384080 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3993488938 0 0) 0x7fb9a415dbd0 con 0x7fb9a41606b0 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a27fc640 1 -- 172.21.15.46:0/1054384080 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb990003520 con 0x7fb9a41606b0 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 -- 172.21.15.46:0/1054384080 >> v2:172.21.15.78:3300/0 conn(0x7fb9a41602d0 msgr2=0x7fb9a40cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 --2- 172.21.15.46:0/1054384080 >> v2:172.21.15.78:3300/0 conn(0x7fb9a41602d0 0x7fb9a40cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 -- 172.21.15.46:0/1054384080 >> v1:172.21.15.78:6789/0 conn(0x7fb9a41606b0 legacy=0x7fb9a40d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a37fe640 1 -- 172.21.15.46:0/1054384080 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a40cdc60 con 0x7fb9a409ab40 2024-04-15T15:16:02.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.111+0000 7fb9a2ffd640 1 --2- 172.21.15.46:0/1054384080 >> v2:172.21.15.78:3300/0 conn(0x7fb9a41602d0 0x7fb9a40cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.112+0000 7fb9a37fe640 1 --2- 172.21.15.46:0/1054384080 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a409afd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009e50 tx=0x7fb99c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9af7e331a51f31c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.112+0000 7fb9a27fc640 1 -- 172.21.15.46:0/1054384080 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c02cb00 con 0x7fb9a409ab40 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.112+0000 7fb9a27fc640 1 -- 172.21.15.46:0/1054384080 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99c02cca0 con 0x7fb9a409ab40 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.112+0000 7fb9a27fc640 1 -- 172.21.15.46:0/1054384080 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c034610 con 0x7fb9a409ab40 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 -- 172.21.15.46:0/1054384080 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 msgr2=0x7fb9a409afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 --2- 172.21.15.46:0/1054384080 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a409afd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009e50 tx=0x7fb99c02bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 -- 172.21.15.46:0/1054384080 shutdown_connections 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 --2- 172.21.15.46:0/1054384080 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a409afd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 --2- 172.21.15.46:0/1054384080 >> v2:172.21.15.78:3300/0 conn(0x7fb9a41602d0 0x7fb9a40cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 -- 172.21.15.46:0/1054384080 >> 172.21.15.46:0/1054384080 conn(0x7fb9a40d2460 msgr2=0x7fb9a40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 -- 172.21.15.46:0/1054384080 shutdown_connections 2024-04-15T15:16:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.113+0000 7fb9a99af640 1 -- 172.21.15.46:0/1054384080 wait complete. 2024-04-15T15:16:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.114+0000 7fb9a99af640 1 Processor -- start 2024-04-15T15:16:02.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.114+0000 7fb9a99af640 1 -- start start 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a99af640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a406e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a99af640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a41602d0 0x7fb9a40c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a99af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 0x7fb9a40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a99af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a4160b10 con 0x7fb9a41606b0 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a99af640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a40ce220 con 0x7fb9a41602d0 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a99af640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a409b510 con 0x7fb9a409ab40 2024-04-15T15:16:02.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 0x7fb9a40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 0x7fb9a40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41996/0 (socket says 172.21.15.46:41996) 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 -- 172.21.15.46:0/4240213017 learned_addr learned my addr 172.21.15.46:0/4240213017 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a2ffd640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a406e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb99bfff640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a41602d0 0x7fb9a40c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 -- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 msgr2=0x7fb9a406e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a406e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 -- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a41602d0 msgr2=0x7fb9a40c4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a41602d0 0x7fb9a40c4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.115+0000 7fb9a37fe640 1 -- 172.21.15.46:0/4240213017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a406eb10 con 0x7fb9a41606b0 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.116+0000 7fb9a37fe640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 0x7fb9a40c51e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009e50 tx=0x7fb99c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.116+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c043020 con 0x7fb9a41606b0 2024-04-15T15:16:02.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.116+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99c02ce50 con 0x7fb9a41606b0 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.116+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a40b7ae0 con 0x7fb9a41606b0 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.116+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c009510 con 0x7fb9a41606b0 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.116+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a41f5920 con 0x7fb9a41606b0 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.117+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a40689e0 con 0x7fb9a41606b0 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.117+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb99c046020 con 0x7fb9a41606b0 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.118+0000 7fb9a0ff9640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb98406c8f0 0x7fb98406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.118+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb99c0ae250 con 0x7fb9a41606b0 2024-04-15T15:16:02.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.118+0000 7fb9a2ffd640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb98406c8f0 0x7fb98406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.118+0000 7fb9a2ffd640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb98406c8f0 0x7fb98406edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb994009560 tx=0x7fb994009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.122+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb99c04d030 con 0x7fb9a41606b0 2024-04-15T15:16:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.307+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9a40b22f0 con 0x7fb9a41606b0 2024-04-15T15:16:02.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.308+0000 7fb9a0ff9640 1 -- 172.21.15.46:0/4240213017 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fb99c07bcf0 con 0x7fb9a41606b0 2024-04-15T15:16:02.310 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:02.310 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb98406c8f0 msgr2=0x7fb98406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb98406c8f0 0x7fb98406edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb994009560 tx=0x7fb994009190 comp rx=0 tx=0).stop 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 msgr2=0x7fb9a40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 0x7fb9a40c51e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb99c009e50 tx=0x7fb99c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 shutdown_connections 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb98406c8f0 0x7fb98406edb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb9a41606b0 0x7fb9a40c51e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9a41602d0 0x7fb9a40c4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 --2- 172.21.15.46:0/4240213017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb9a409ab40 0x7fb9a406e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 >> 172.21.15.46:0/4240213017 conn(0x7fb9a40d2460 msgr2=0x7fb9a40b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 shutdown_connections 2024-04-15T15:16:02.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.310+0000 7fb9a99af640 1 -- 172.21.15.46:0/4240213017 wait complete. 2024-04-15T15:16:02.323 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:16:02.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:02.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.426+0000 7f9a0de63640 1 Processor -- start 2024-04-15T15:16:02.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.427+0000 7f9a0de63640 1 -- start start 2024-04-15T15:16:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.427+0000 7f9a0de63640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9a0815e790 0x7f9a080b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.427+0000 7f9a0de63640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0815eb70 con 0x7f9a080b7be0 2024-04-15T15:16:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a0de63640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a080d3c80 con 0x7f9a080d1cf0 2024-04-15T15:16:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a0de63640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f9a080d3790 con 0x7f9a0815e790 2024-04-15T15:16:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a06ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9a080b7be0 0x7f9a080d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55560/0 (socket says 172.21.15.46:55560) 2024-04-15T15:16:02.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a06ffd640 1 -- 172.21.15.46:0/393455171 learned_addr learned my addr 172.21.15.46:0/393455171 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:02.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a077fe640 1 --2- 172.21.15.46:0/393455171 >> v2:172.21.15.78:3300/0 conn(0x7f9a0815e790 0x7f9a080b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a067fc640 1 -- 172.21.15.46:0/393455171 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872647523 0 0) 0x7f9a0815eb70 con 0x7f9a080b7be0 2024-04-15T15:16:02.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a067fc640 1 -- 172.21.15.46:0/393455171 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99f4003520 con 0x7f9a080b7be0 2024-04-15T15:16:02.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a077fe640 1 -- 172.21.15.46:0/393455171 >> v1:172.21.15.78:6789/0 conn(0x7f9a080d1cf0 legacy=0x7f9a080d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a077fe640 1 -- 172.21.15.46:0/393455171 >> v1:172.21.15.46:6789/0 conn(0x7f9a080b7be0 legacy=0x7f9a080d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.428+0000 7f9a077fe640 1 -- 172.21.15.46:0/393455171 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a080d3f60 con 0x7f9a0815e790 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.429+0000 7f9a077fe640 1 --2- 172.21.15.46:0/393455171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a080b76a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f99f80097a0 tx=0x7f99f802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aafcdecf531f116a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.429+0000 7f9a067fc640 1 -- 172.21.15.46:0/393455171 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f80315a0 con 0x7f9a0815e790 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.429+0000 7f9a067fc640 1 -- 172.21.15.46:0/393455171 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f8037070 con 0x7f9a0815e790 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.429+0000 7f9a067fc640 1 -- 172.21.15.46:0/393455171 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8031940 con 0x7f9a0815e790 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 -- 172.21.15.46:0/393455171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 msgr2=0x7f9a080b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 --2- 172.21.15.46:0/393455171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a080b76a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f99f80097a0 tx=0x7f99f802b840 comp rx=0 tx=0).stop 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 -- 172.21.15.46:0/393455171 shutdown_connections 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 --2- 172.21.15.46:0/393455171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a080b76a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 -- 172.21.15.46:0/393455171 >> 172.21.15.46:0/393455171 conn(0x7f9a08067b40 msgr2=0x7f9a08067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 -- 172.21.15.46:0/393455171 shutdown_connections 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.430+0000 7f9a0de63640 1 -- 172.21.15.46:0/393455171 wait complete. 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.431+0000 7f9a0de63640 1 Processor -- start 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.431+0000 7f9a0de63640 1 -- start start 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a0de63640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a080b7be0 0x7f9a081f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a0de63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 0x7f9a080b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a0de63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a0814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a0de63640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9a080d5150 con 0x7f9a080d1cf0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a0de63640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9a080d45b0 con 0x7f9a0815e790 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a0de63640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9a080d52d0 con 0x7f9a080b7be0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 0x7f9a080b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 0x7f9a080b8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42012/0 (socket says 172.21.15.46:42012) 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 -- 172.21.15.46:0/1019475534 learned_addr learned my addr 172.21.15.46:0/1019475534 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a077fe640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a080b7be0 0x7f9a081f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a07fff640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a0814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 -- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a080b7be0 msgr2=0x7f9a081f84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a080b7be0 0x7f9a081f84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 -- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 msgr2=0x7f9a0814f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a0814f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 -- 172.21.15.46:0/1019475534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a080c93e0 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.432+0000 7f9a06ffd640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 0x7f9a080b8310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c430 tx=0x7f99f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.433+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f0017070 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.433+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f0005ce0 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.433+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a080abe00 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.433+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00072f0 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.433+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a080c9960 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.434+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a0815d220 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.434+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f99f001a070 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.435+0000 7f99fffff640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f99e806c8f0 0x7f99e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.435+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f99f008b1d0 con 0x7f9a080d1cf0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.435+0000 7f9a077fe640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f99e806c8f0 0x7f99e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.435+0000 7f9a077fe640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f99e806c8f0 0x7f99e806edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f99f8036000 tx=0x7f99f8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.439+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99f0058d20 con 0x7f9a080d1cf0 2024-04-15T15:16:02.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.623+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9a080689e0 con 0x7f99e806c8f0 2024-04-15T15:16:02.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.632+0000 7f99fffff640 1 -- 172.21.15.46:0/1019475534 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f9a080689e0 con 0x7f99e806c8f0 2024-04-15T15:16:02.643 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:02.645 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f99e806c8f0 msgr2=0x7f99e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f99e806c8f0 0x7f99e806edb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f99f8036000 tx=0x7f99f8008030 comp rx=0 tx=0).stop 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 msgr2=0x7f9a080b8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 0x7f9a080b8310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f99f000c430 tx=0x7f99f000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 shutdown_connections 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f99e806c8f0 0x7f99e806edb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a0815e790 0x7f9a0814f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a080d1cf0 0x7f9a080b8310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 --2- 172.21.15.46:0/1019475534 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9a080b7be0 0x7f9a081f84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 >> 172.21.15.46:0/1019475534 conn(0x7f9a08067b40 msgr2=0x7f9a080b9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 shutdown_connections 2024-04-15T15:16:02.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.636+0000 7f9a0de63640 1 -- 172.21.15.46:0/1019475534 wait complete. 2024-04-15T15:16:02.650 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-15T15:16:02.358856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231928,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742792,"statfs":{"total":773094113280,"available":772856619008,"internally_reserved":0,"allocated":3350528,"data_stored":2022712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003396"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439492+0000","last_change":"2024-04-15T15:15:37.566709+0000","last_active":"2024-04-15T15:15:44.439492+0000","last_peered":"2024-04-15T15:15:44.439492+0000","last_clean":"2024-04-15T15:15:44.439492+0000","last_became_active":"2024-04-15T15:15:37.566073+0000","last_became_peered":"2024-04-15T15:15:37.566073+0000","last_unstale":"2024-04-15T15:15:44.439492+0000","last_undegraded":"2024-04-15T15:15:44.439492+0000","last_fullsized":"2024-04-15T15:15:44.439492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:40:55.907098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440303+0000","last_change":"2024-04-15T15:15:38.565494+0000","last_active":"2024-04-15T15:15:44.440303+0000","last_peered":"2024-04-15T15:15:44.440303+0000","last_clean":"2024-04-15T15:15:44.440303+0000","last_became_active":"2024-04-15T15:15:38.563911+0000","last_became_peered":"2024-04-15T15:15:38.563911+0000","last_unstale":"2024-04-15T15:15:44.440303+0000","last_undegraded":"2024-04-15T15:15:44.440303+0000","last_fullsized":"2024-04-15T15:15:44.440303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439886+0000","last_change":"2024-04-15T15:15:42.661119+0000","last_active":"2024-04-15T15:15:44.439886+0000","last_peered":"2024-04-15T15:15:44.439886+0000","last_clean":"2024-04-15T15:15:44.439886+0000","last_became_active":"2024-04-15T15:15:42.660440+0000","last_became_peered":"2024-04-15T15:15:42.660440+0000","last_unstale":"2024-04-15T15:15:44.439886+0000","last_undegraded":"2024-04-15T15:15:44.439886+0000","last_fullsized":"2024-04-15T15:15:44.439886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440400+0000","last_change":"2024-04-15T15:15:37.563320+0000","last_active":"2024-04-15T15:15:44.440400+0000","last_peered":"2024-04-15T15:15:44.440400+0000","last_clean":"2024-04-15T15:15:44.440400+0000","last_became_active":"2024-04-15T15:15:37.563058+0000","last_became_peered":"2024-04-15T15:15:37.563058+0000","last_unstale":"2024-04-15T15:15:44.440400+0000","last_undegraded":"2024-04-15T15:15:44.440400+0000","last_fullsized":"2024-04-15T15:15:44.440400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439174+0000","last_change":"2024-04-15T15:15:38.573022+0000","last_active":"2024-04-15T15:15:44.439174+0000","last_peered":"2024-04-15T15:15:44.439174+0000","last_clean":"2024-04-15T15:15:44.439174+0000","last_became_active":"2024-04-15T15:15:38.572726+0000","last_became_peered":"2024-04-15T15:15:38.572726+0000","last_unstale":"2024-04-15T15:15:44.439174+0000","last_undegraded":"2024-04-15T15:15:44.439174+0000","last_fullsized":"2024-04-15T15:15:44.439174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587093+0000","last_change":"2024-04-15T15:15:42.590697+0000","last_active":"2024-04-15T15:15:43.587093+0000","last_peered":"2024-04-15T15:15:43.587093+0000","last_clean":"2024-04-15T15:15:43.587093+0000","last_became_active":"2024-04-15T15:15:42.590272+0000","last_became_peered":"2024-04-15T15:15:42.590272+0000","last_unstale":"2024-04-15T15:15:43.587093+0000","last_undegraded":"2024-04-15T15:15:43.587093+0000","last_fullsized":"2024-04-15T15:15:43.587093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:10.918812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439863+0000","last_change":"2024-04-15T15:15:37.564953+0000","last_active":"2024-04-15T15:15:44.439863+0000","last_peered":"2024-04-15T15:15:44.439863+0000","last_clean":"2024-04-15T15:15:44.439863+0000","last_became_active":"2024-04-15T15:15:37.564653+0000","last_became_peered":"2024-04-15T15:15:37.564653+0000","last_unstale":"2024-04-15T15:15:44.439863+0000","last_undegraded":"2024-04-15T15:15:44.439863+0000","last_fullsized":"2024-04-15T15:15:44.439863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440820+0000","last_change":"2024-04-15T15:15:38.568986+0000","last_active":"2024-04-15T15:15:44.440820+0000","last_peered":"2024-04-15T15:15:44.440820+0000","last_clean":"2024-04-15T15:15:44.440820+0000","last_became_active":"2024-04-15T15:15:38.566559+0000","last_became_peered":"2024-04-15T15:15:38.566559+0000","last_unstale":"2024-04-15T15:15:44.440820+0000","last_undegraded":"2024-04-15T15:15:44.440820+0000","last_fullsized":"2024-04-15T15:15:44.440820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587815+0000","last_change":"2024-04-15T15:15:42.592706+0000","last_active":"2024-04-15T15:15:43.587815+0000","last_peered":"2024-04-15T15:15:43.587815+0000","last_clean":"2024-04-15T15:15:43.587815+0000","last_became_active":"2024-04-15T15:15:42.592134+0000","last_became_peered":"2024-04-15T15:15:42.592134+0000","last_unstale":"2024-04-15T15:15:43.587815+0000","last_undegraded":"2024-04-15T15:15:43.587815+0000","last_fullsized":"2024-04-15T15:15:43.587815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:41.894568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439525+0000","last_change":"2024-04-15T15:15:37.567857+0000","last_active":"2024-04-15T15:15:44.439525+0000","last_peered":"2024-04-15T15:15:44.439525+0000","last_clean":"2024-04-15T15:15:44.439525+0000","last_became_active":"2024-04-15T15:15:37.567560+0000","last_became_peered":"2024-04-15T15:15:37.567560+0000","last_unstale":"2024-04-15T15:15:44.439525+0000","last_undegraded":"2024-04-15T15:15:44.439525+0000","last_fullsized":"2024-04-15T15:15:44.439525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:36:21.001555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440328+0000","last_change":"2024-04-15T15:15:38.568788+0000","last_active":"2024-04-15T15:15:44.440328+0000","last_peered":"2024-04-15T15:15:44.440328+0000","last_clean":"2024-04-15T15:15:44.440328+0000","last_became_active":"2024-04-15T15:15:38.567501+0000","last_became_peered":"2024-04-15T15:15:38.567501+0000","last_unstale":"2024-04-15T15:15:44.440328+0000","last_undegraded":"2024-04-15T15:15:44.440328+0000","last_fullsized":"2024-04-15T15:15:44.440328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587164+0000","last_change":"2024-04-15T15:15:42.587928+0000","last_active":"2024-04-15T15:15:43.587164+0000","last_peered":"2024-04-15T15:15:43.587164+0000","last_clean":"2024-04-15T15:15:43.587164+0000","last_became_active":"2024-04-15T15:15:42.587518+0000","last_became_peered":"2024-04-15T15:15:42.587518+0000","last_unstale":"2024-04-15T15:15:43.587164+0000","last_undegraded":"2024-04-15T15:15:43.587164+0000","last_fullsized":"2024-04-15T15:15:43.587164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:50.908590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587148+0000","last_change":"2024-04-15T15:15:38.560568+0000","last_active":"2024-04-15T15:15:43.587148+0000","last_peered":"2024-04-15T15:15:43.587148+0000","last_clean":"2024-04-15T15:15:43.587148+0000","last_became_active":"2024-04-15T15:15:38.560292+0000","last_became_peered":"2024-04-15T15:15:38.560292+0000","last_unstale":"2024-04-15T15:15:43.587148+0000","last_undegraded":"2024-04-15T15:15:43.587148+0000","last_fullsized":"2024-04-15T15:15:43.587148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440210+0000","last_change":"2024-04-15T15:15:37.570166+0000","last_active":"2024-04-15T15:15:44.440210+0000","last_peered":"2024-04-15T15:15:44.440210+0000","last_clean":"2024-04-15T15:15:44.440210+0000","last_became_active":"2024-04-15T15:15:37.569862+0000","last_became_peered":"2024-04-15T15:15:37.569862+0000","last_unstale":"2024-04-15T15:15:44.440210+0000","last_undegraded":"2024-04-15T15:15:44.440210+0000","last_fullsized":"2024-04-15T15:15:44.440210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:55:13.182192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588794+0000","last_change":"2024-04-15T15:15:42.590929+0000","last_active":"2024-04-15T15:15:43.588794+0000","last_peered":"2024-04-15T15:15:43.588794+0000","last_clean":"2024-04-15T15:15:43.588794+0000","last_became_active":"2024-04-15T15:15:42.589759+0000","last_became_peered":"2024-04-15T15:15:42.589759+0000","last_unstale":"2024-04-15T15:15:43.588794+0000","last_undegraded":"2024-04-15T15:15:43.588794+0000","last_fullsized":"2024-04-15T15:15:43.588794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439955+0000","last_change":"2024-04-15T15:15:38.563606+0000","last_active":"2024-04-15T15:15:44.439955+0000","last_peered":"2024-04-15T15:15:44.439955+0000","last_clean":"2024-04-15T15:15:44.439955+0000","last_became_active":"2024-04-15T15:15:38.562885+0000","last_became_peered":"2024-04-15T15:15:38.562885+0000","last_unstale":"2024-04-15T15:15:44.439955+0000","last_undegraded":"2024-04-15T15:15:44.439955+0000","last_fullsized":"2024-04-15T15:15:44.439955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587327+0000","last_change":"2024-04-15T15:15:37.562063+0000","last_active":"2024-04-15T15:15:43.587327+0000","last_peered":"2024-04-15T15:15:43.587327+0000","last_clean":"2024-04-15T15:15:43.587327+0000","last_became_active":"2024-04-15T15:15:37.561757+0000","last_became_peered":"2024-04-15T15:15:37.561757+0000","last_unstale":"2024-04-15T15:15:43.587327+0000","last_undegraded":"2024-04-15T15:15:43.587327+0000","last_fullsized":"2024-04-15T15:15:43.587327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588354+0000","last_change":"2024-04-15T15:15:42.591439+0000","last_active":"2024-04-15T15:15:43.588354+0000","last_peered":"2024-04-15T15:15:43.588354+0000","last_clean":"2024-04-15T15:15:43.588354+0000","last_became_active":"2024-04-15T15:15:42.591030+0000","last_became_peered":"2024-04-15T15:15:42.591030+0000","last_unstale":"2024-04-15T15:15:43.588354+0000","last_undegraded":"2024-04-15T15:15:43.588354+0000","last_fullsized":"2024-04-15T15:15:43.588354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:19.368841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439666+0000","last_change":"2024-04-15T15:15:38.566770+0000","last_active":"2024-04-15T15:15:44.439666+0000","last_peered":"2024-04-15T15:15:44.439666+0000","last_clean":"2024-04-15T15:15:44.439666+0000","last_became_active":"2024-04-15T15:15:38.566484+0000","last_became_peered":"2024-04-15T15:15:38.566484+0000","last_unstale":"2024-04-15T15:15:44.439666+0000","last_undegraded":"2024-04-15T15:15:44.439666+0000","last_fullsized":"2024-04-15T15:15:44.439666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586733+0000","last_change":"2024-04-15T15:15:37.559565+0000","last_active":"2024-04-15T15:15:43.586733+0000","last_peered":"2024-04-15T15:15:43.586733+0000","last_clean":"2024-04-15T15:15:43.586733+0000","last_became_active":"2024-04-15T15:15:37.558121+0000","last_became_peered":"2024-04-15T15:15:37.558121+0000","last_unstale":"2024-04-15T15:15:43.586733+0000","last_undegraded":"2024-04-15T15:15:43.586733+0000","last_fullsized":"2024-04-15T15:15:43.586733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.655888+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.655425+0000","last_became_peered":"2024-04-15T15:15:42.655425+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587565+0000","last_change":"2024-04-15T15:15:38.558135+0000","last_active":"2024-04-15T15:15:43.587565+0000","last_peered":"2024-04-15T15:15:43.587565+0000","last_clean":"2024-04-15T15:15:43.587565+0000","last_became_active":"2024-04-15T15:15:38.557849+0000","last_became_peered":"2024-04-15T15:15:38.557849+0000","last_unstale":"2024-04-15T15:15:43.587565+0000","last_undegraded":"2024-04-15T15:15:43.587565+0000","last_fullsized":"2024-04-15T15:15:43.587565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:08.649907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439485+0000","last_change":"2024-04-15T15:15:37.568252+0000","last_active":"2024-04-15T15:15:44.439485+0000","last_peered":"2024-04-15T15:15:44.439485+0000","last_clean":"2024-04-15T15:15:44.439485+0000","last_became_active":"2024-04-15T15:15:37.567982+0000","last_became_peered":"2024-04-15T15:15:37.567982+0000","last_unstale":"2024-04-15T15:15:44.439485+0000","last_undegraded":"2024-04-15T15:15:44.439485+0000","last_fullsized":"2024-04-15T15:15:44.439485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588452+0000","last_change":"2024-04-15T15:15:42.593489+0000","last_active":"2024-04-15T15:15:43.588452+0000","last_peered":"2024-04-15T15:15:43.588452+0000","last_clean":"2024-04-15T15:15:43.588452+0000","last_became_active":"2024-04-15T15:15:42.593152+0000","last_became_peered":"2024-04-15T15:15:42.593152+0000","last_unstale":"2024-04-15T15:15:43.588452+0000","last_undegraded":"2024-04-15T15:15:43.588452+0000","last_fullsized":"2024-04-15T15:15:43.588452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439610+0000","last_change":"2024-04-15T15:15:38.559585+0000","last_active":"2024-04-15T15:15:44.439610+0000","last_peered":"2024-04-15T15:15:44.439610+0000","last_clean":"2024-04-15T15:15:44.439610+0000","last_became_active":"2024-04-15T15:15:38.559225+0000","last_became_peered":"2024-04-15T15:15:38.559225+0000","last_unstale":"2024-04-15T15:15:44.439610+0000","last_undegraded":"2024-04-15T15:15:44.439610+0000","last_fullsized":"2024-04-15T15:15:44.439610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.589097+0000","last_change":"2024-04-15T15:15:37.558650+0000","last_active":"2024-04-15T15:15:43.589097+0000","last_peered":"2024-04-15T15:15:43.589097+0000","last_clean":"2024-04-15T15:15:43.589097+0000","last_became_active":"2024-04-15T15:15:37.558128+0000","last_became_peered":"2024-04-15T15:15:37.558128+0000","last_unstale":"2024-04-15T15:15:43.589097+0000","last_undegraded":"2024-04-15T15:15:43.589097+0000","last_fullsized":"2024-04-15T15:15:43.589097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587388+0000","last_change":"2024-04-15T15:15:42.655806+0000","last_active":"2024-04-15T15:15:43.587388+0000","last_peered":"2024-04-15T15:15:43.587388+0000","last_clean":"2024-04-15T15:15:43.587388+0000","last_became_active":"2024-04-15T15:15:42.655431+0000","last_became_peered":"2024-04-15T15:15:42.655431+0000","last_unstale":"2024-04-15T15:15:43.587388+0000","last_undegraded":"2024-04-15T15:15:43.587388+0000","last_fullsized":"2024-04-15T15:15:43.587388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:10.032080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440420+0000","last_change":"2024-04-15T15:15:38.567366+0000","last_active":"2024-04-15T15:15:44.440420+0000","last_peered":"2024-04-15T15:15:44.440420+0000","last_clean":"2024-04-15T15:15:44.440420+0000","last_became_active":"2024-04-15T15:15:38.567064+0000","last_became_peered":"2024-04-15T15:15:38.567064+0000","last_unstale":"2024-04-15T15:15:44.440420+0000","last_undegraded":"2024-04-15T15:15:44.440420+0000","last_fullsized":"2024-04-15T15:15:44.440420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587807+0000","last_change":"2024-04-15T15:15:37.554351+0000","last_active":"2024-04-15T15:15:43.587807+0000","last_peered":"2024-04-15T15:15:43.587807+0000","last_clean":"2024-04-15T15:15:43.587807+0000","last_became_active":"2024-04-15T15:15:37.554062+0000","last_became_peered":"2024-04-15T15:15:37.554062+0000","last_unstale":"2024-04-15T15:15:43.587807+0000","last_undegraded":"2024-04-15T15:15:43.587807+0000","last_fullsized":"2024-04-15T15:15:43.587807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438909+0000","last_change":"2024-04-15T15:15:42.659938+0000","last_active":"2024-04-15T15:15:44.438909+0000","last_peered":"2024-04-15T15:15:44.438909+0000","last_clean":"2024-04-15T15:15:44.438909+0000","last_became_active":"2024-04-15T15:15:42.659252+0000","last_became_peered":"2024-04-15T15:15:42.659252+0000","last_unstale":"2024-04-15T15:15:44.438909+0000","last_undegraded":"2024-04-15T15:15:44.438909+0000","last_fullsized":"2024-04-15T15:15:44.438909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:48.210859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587105+0000","last_change":"2024-04-15T15:15:38.561075+0000","last_active":"2024-04-15T15:15:43.587105+0000","last_peered":"2024-04-15T15:15:43.587105+0000","last_clean":"2024-04-15T15:15:43.587105+0000","last_became_active":"2024-04-15T15:15:38.559163+0000","last_became_peered":"2024-04-15T15:15:38.559163+0000","last_unstale":"2024-04-15T15:15:43.587105+0000","last_undegraded":"2024-04-15T15:15:43.587105+0000","last_fullsized":"2024-04-15T15:15:43.587105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440117+0000","last_change":"2024-04-15T15:15:37.568261+0000","last_active":"2024-04-15T15:15:44.440117+0000","last_peered":"2024-04-15T15:15:44.440117+0000","last_clean":"2024-04-15T15:15:44.440117+0000","last_became_active":"2024-04-15T15:15:37.567997+0000","last_became_peered":"2024-04-15T15:15:37.567997+0000","last_unstale":"2024-04-15T15:15:44.440117+0000","last_undegraded":"2024-04-15T15:15:44.440117+0000","last_fullsized":"2024-04-15T15:15:44.440117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:47:24.543446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:42.658160+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:42.657766+0000","last_became_peered":"2024-04-15T15:15:42.657766+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440434+0000","last_change":"2024-04-15T15:15:38.563836+0000","last_active":"2024-04-15T15:15:44.440434+0000","last_peered":"2024-04-15T15:15:44.440434+0000","last_clean":"2024-04-15T15:15:44.440434+0000","last_became_active":"2024-04-15T15:15:38.563487+0000","last_became_peered":"2024-04-15T15:15:38.563487+0000","last_unstale":"2024-04-15T15:15:44.440434+0000","last_undegraded":"2024-04-15T15:15:44.440434+0000","last_fullsized":"2024-04-15T15:15:44.440434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587767+0000","last_change":"2024-04-15T15:15:37.563786+0000","last_active":"2024-04-15T15:15:43.587767+0000","last_peered":"2024-04-15T15:15:43.587767+0000","last_clean":"2024-04-15T15:15:43.587767+0000","last_became_active":"2024-04-15T15:15:37.563491+0000","last_became_peered":"2024-04-15T15:15:37.563491+0000","last_unstale":"2024-04-15T15:15:43.587767+0000","last_undegraded":"2024-04-15T15:15:43.587767+0000","last_fullsized":"2024-04-15T15:15:43.587767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586975+0000","last_change":"2024-04-15T15:15:42.593439+0000","last_active":"2024-04-15T15:15:43.586975+0000","last_peered":"2024-04-15T15:15:43.586975+0000","last_clean":"2024-04-15T15:15:43.586975+0000","last_became_active":"2024-04-15T15:15:42.591555+0000","last_became_peered":"2024-04-15T15:15:42.591555+0000","last_unstale":"2024-04-15T15:15:43.586975+0000","last_undegraded":"2024-04-15T15:15:43.586975+0000","last_fullsized":"2024-04-15T15:15:43.586975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588422+0000","last_change":"2024-04-15T15:15:38.560053+0000","last_active":"2024-04-15T15:15:43.588422+0000","last_peered":"2024-04-15T15:15:43.588422+0000","last_clean":"2024-04-15T15:15:43.588422+0000","last_became_active":"2024-04-15T15:15:38.559462+0000","last_became_peered":"2024-04-15T15:15:38.559462+0000","last_unstale":"2024-04-15T15:15:43.588422+0000","last_undegraded":"2024-04-15T15:15:43.588422+0000","last_fullsized":"2024-04-15T15:15:43.588422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:50.427117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440424+0000","last_change":"2024-04-15T15:15:37.561463+0000","last_active":"2024-04-15T15:15:44.440424+0000","last_peered":"2024-04-15T15:15:44.440424+0000","last_clean":"2024-04-15T15:15:44.440424+0000","last_became_active":"2024-04-15T15:15:37.560849+0000","last_became_peered":"2024-04-15T15:15:37.560849+0000","last_unstale":"2024-04-15T15:15:44.440424+0000","last_undegraded":"2024-04-15T15:15:44.440424+0000","last_fullsized":"2024-04-15T15:15:44.440424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.594805+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.594322+0000","last_became_peered":"2024-04-15T15:15:42.594322+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586925+0000","last_change":"2024-04-15T15:15:38.559721+0000","last_active":"2024-04-15T15:15:43.586925+0000","last_peered":"2024-04-15T15:15:43.586925+0000","last_clean":"2024-04-15T15:15:43.586925+0000","last_became_active":"2024-04-15T15:15:38.559440+0000","last_became_peered":"2024-04-15T15:15:38.559440+0000","last_unstale":"2024-04-15T15:15:43.586925+0000","last_undegraded":"2024-04-15T15:15:43.586925+0000","last_fullsized":"2024-04-15T15:15:43.586925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769205+0000","last_change":"2024-04-15T15:15:37.562651+0000","last_active":"2024-04-15T15:15:43.769205+0000","last_peered":"2024-04-15T15:15:43.769205+0000","last_clean":"2024-04-15T15:15:43.769205+0000","last_became_active":"2024-04-15T15:15:37.562267+0000","last_became_peered":"2024-04-15T15:15:37.562267+0000","last_unstale":"2024-04-15T15:15:43.769205+0000","last_undegraded":"2024-04-15T15:15:43.769205+0000","last_fullsized":"2024-04-15T15:15:43.769205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587782+0000","last_change":"2024-04-15T15:15:42.595294+0000","last_active":"2024-04-15T15:15:43.587782+0000","last_peered":"2024-04-15T15:15:43.587782+0000","last_clean":"2024-04-15T15:15:43.587782+0000","last_became_active":"2024-04-15T15:15:42.594395+0000","last_became_peered":"2024-04-15T15:15:42.594395+0000","last_unstale":"2024-04-15T15:15:43.587782+0000","last_undegraded":"2024-04-15T15:15:43.587782+0000","last_fullsized":"2024-04-15T15:15:43.587782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438893+0000","last_change":"2024-04-15T15:15:38.566568+0000","last_active":"2024-04-15T15:15:44.438893+0000","last_peered":"2024-04-15T15:15:44.438893+0000","last_clean":"2024-04-15T15:15:44.438893+0000","last_became_active":"2024-04-15T15:15:38.566261+0000","last_became_peered":"2024-04-15T15:15:38.566261+0000","last_unstale":"2024-04-15T15:15:44.438893+0000","last_undegraded":"2024-04-15T15:15:44.438893+0000","last_fullsized":"2024-04-15T15:15:44.438893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439388+0000","last_change":"2024-04-15T15:15:37.567337+0000","last_active":"2024-04-15T15:15:44.439388+0000","last_peered":"2024-04-15T15:15:44.439388+0000","last_clean":"2024-04-15T15:15:44.439388+0000","last_became_active":"2024-04-15T15:15:37.567042+0000","last_became_peered":"2024-04-15T15:15:37.567042+0000","last_unstale":"2024-04-15T15:15:44.439388+0000","last_undegraded":"2024-04-15T15:15:44.439388+0000","last_fullsized":"2024-04-15T15:15:44.439388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439562+0000","last_change":"2024-04-15T15:15:42.657091+0000","last_active":"2024-04-15T15:15:44.439562+0000","last_peered":"2024-04-15T15:15:44.439562+0000","last_clean":"2024-04-15T15:15:44.439562+0000","last_became_active":"2024-04-15T15:15:42.656638+0000","last_became_peered":"2024-04-15T15:15:42.656638+0000","last_unstale":"2024-04-15T15:15:44.439562+0000","last_undegraded":"2024-04-15T15:15:44.439562+0000","last_fullsized":"2024-04-15T15:15:44.439562+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:46.336878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439991+0000","last_change":"2024-04-15T15:15:38.576736+0000","last_active":"2024-04-15T15:15:44.439991+0000","last_peered":"2024-04-15T15:15:44.439991+0000","last_clean":"2024-04-15T15:15:44.439991+0000","last_became_active":"2024-04-15T15:15:38.574326+0000","last_became_peered":"2024-04-15T15:15:38.574326+0000","last_unstale":"2024-04-15T15:15:44.439991+0000","last_undegraded":"2024-04-15T15:15:44.439991+0000","last_fullsized":"2024-04-15T15:15:44.439991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440769+0000","last_change":"2024-04-15T15:15:37.557351+0000","last_active":"2024-04-15T15:15:44.440769+0000","last_peered":"2024-04-15T15:15:44.440769+0000","last_clean":"2024-04-15T15:15:44.440769+0000","last_became_active":"2024-04-15T15:15:37.557036+0000","last_became_peered":"2024-04-15T15:15:37.557036+0000","last_unstale":"2024-04-15T15:15:44.440769+0000","last_undegraded":"2024-04-15T15:15:44.440769+0000","last_fullsized":"2024-04-15T15:15:44.440769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439564+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:15:44.439564+0000","last_peered":"2024-04-15T15:15:44.439564+0000","last_clean":"2024-04-15T15:15:44.439564+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:15:44.439564+0000","last_undegraded":"2024-04-15T15:15:44.439564+0000","last_fullsized":"2024-04-15T15:15:44.439564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588337+0000","last_change":"2024-04-15T15:15:42.658428+0000","last_active":"2024-04-15T15:15:43.588337+0000","last_peered":"2024-04-15T15:15:43.588337+0000","last_clean":"2024-04-15T15:15:43.588337+0000","last_became_active":"2024-04-15T15:15:42.658014+0000","last_became_peered":"2024-04-15T15:15:42.658014+0000","last_unstale":"2024-04-15T15:15:43.588337+0000","last_undegraded":"2024-04-15T15:15:43.588337+0000","last_fullsized":"2024-04-15T15:15:43.588337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439749+0000","last_change":"2024-04-15T15:15:38.572859+0000","last_active":"2024-04-15T15:15:44.439749+0000","last_peered":"2024-04-15T15:15:44.439749+0000","last_clean":"2024-04-15T15:15:44.439749+0000","last_became_active":"2024-04-15T15:15:38.572580+0000","last_became_peered":"2024-04-15T15:15:38.572580+0000","last_unstale":"2024-04-15T15:15:44.439749+0000","last_undegraded":"2024-04-15T15:15:44.439749+0000","last_fullsized":"2024-04-15T15:15:44.439749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769429+0000","last_change":"2024-04-15T15:15:37.555795+0000","last_active":"2024-04-15T15:15:43.769429+0000","last_peered":"2024-04-15T15:15:43.769429+0000","last_clean":"2024-04-15T15:15:43.769429+0000","last_became_active":"2024-04-15T15:15:37.555502+0000","last_became_peered":"2024-04-15T15:15:37.555502+0000","last_unstale":"2024-04-15T15:15:43.769429+0000","last_undegraded":"2024-04-15T15:15:43.769429+0000","last_fullsized":"2024-04-15T15:15:43.769429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587412+0000","last_change":"2024-04-15T15:15:42.655842+0000","last_active":"2024-04-15T15:15:43.587412+0000","last_peered":"2024-04-15T15:15:43.587412+0000","last_clean":"2024-04-15T15:15:43.587412+0000","last_became_active":"2024-04-15T15:15:42.655470+0000","last_became_peered":"2024-04-15T15:15:42.655470+0000","last_unstale":"2024-04-15T15:15:43.587412+0000","last_undegraded":"2024-04-15T15:15:43.587412+0000","last_fullsized":"2024-04-15T15:15:43.587412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440874+0000","last_change":"2024-04-15T15:15:38.573853+0000","last_active":"2024-04-15T15:15:44.440874+0000","last_peered":"2024-04-15T15:15:44.440874+0000","last_clean":"2024-04-15T15:15:44.440874+0000","last_became_active":"2024-04-15T15:15:38.573585+0000","last_became_peered":"2024-04-15T15:15:38.573585+0000","last_unstale":"2024-04-15T15:15:44.440874+0000","last_undegraded":"2024-04-15T15:15:44.440874+0000","last_fullsized":"2024-04-15T15:15:44.440874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587697+0000","last_change":"2024-04-15T15:15:37.557027+0000","last_active":"2024-04-15T15:15:43.587697+0000","last_peered":"2024-04-15T15:15:43.587697+0000","last_clean":"2024-04-15T15:15:43.587697+0000","last_became_active":"2024-04-15T15:15:37.556725+0000","last_became_peered":"2024-04-15T15:15:37.556725+0000","last_unstale":"2024-04-15T15:15:43.587697+0000","last_undegraded":"2024-04-15T15:15:43.587697+0000","last_fullsized":"2024-04-15T15:15:43.587697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439555+0000","last_change":"2024-04-15T15:15:42.593049+0000","last_active":"2024-04-15T15:15:44.439555+0000","last_peered":"2024-04-15T15:15:44.439555+0000","last_clean":"2024-04-15T15:15:44.439555+0000","last_became_active":"2024-04-15T15:15:42.592494+0000","last_became_peered":"2024-04-15T15:15:42.592494+0000","last_unstale":"2024-04-15T15:15:44.439555+0000","last_undegraded":"2024-04-15T15:15:44.439555+0000","last_fullsized":"2024-04-15T15:15:44.439555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440121+0000","last_change":"2024-04-15T15:15:38.574063+0000","last_active":"2024-04-15T15:15:44.440121+0000","last_peered":"2024-04-15T15:15:44.440121+0000","last_clean":"2024-04-15T15:15:44.440121+0000","last_became_active":"2024-04-15T15:15:38.572802+0000","last_became_peered":"2024-04-15T15:15:38.572802+0000","last_unstale":"2024-04-15T15:15:44.440121+0000","last_undegraded":"2024-04-15T15:15:44.440121+0000","last_fullsized":"2024-04-15T15:15:44.440121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:42.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769497+0000","last_change":"2024-04-15T15:15:37.563114+0000","last_active":"2024-04-15T15:15:43.769497+0000","last_peered":"2024-04-15T15:15:43.769497+0000","last_clean":"2024-04-15T15:15:43.769497+0000","last_became_active":"2024-04-15T15:15:37.562746+0000","last_became_peered":"2024-04-15T15:15:37.562746+0000","last_unstale":"2024-04-15T15:15:43.769497+0000","last_undegraded":"2024-04-15T15:15:43.769497+0000","last_fullsized":"2024-04-15T15:15:43.769497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439239+0000","last_change":"2024-04-15T15:15:42.658786+0000","last_active":"2024-04-15T15:15:44.439239+0000","last_peered":"2024-04-15T15:15:44.439239+0000","last_clean":"2024-04-15T15:15:44.439239+0000","last_became_active":"2024-04-15T15:15:42.658329+0000","last_became_peered":"2024-04-15T15:15:42.658329+0000","last_unstale":"2024-04-15T15:15:44.439239+0000","last_undegraded":"2024-04-15T15:15:44.439239+0000","last_fullsized":"2024-04-15T15:15:44.439239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439930+0000","last_change":"2024-04-15T15:15:38.573032+0000","last_active":"2024-04-15T15:15:44.439930+0000","last_peered":"2024-04-15T15:15:44.439930+0000","last_clean":"2024-04-15T15:15:44.439930+0000","last_became_active":"2024-04-15T15:15:38.572689+0000","last_became_peered":"2024-04-15T15:15:38.572689+0000","last_unstale":"2024-04-15T15:15:44.439930+0000","last_undegraded":"2024-04-15T15:15:44.439930+0000","last_fullsized":"2024-04-15T15:15:44.439930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588533+0000","last_change":"2024-04-15T15:15:37.565326+0000","last_active":"2024-04-15T15:15:43.588533+0000","last_peered":"2024-04-15T15:15:43.588533+0000","last_clean":"2024-04-15T15:15:43.588533+0000","last_became_active":"2024-04-15T15:15:37.564993+0000","last_became_peered":"2024-04-15T15:15:37.564993+0000","last_unstale":"2024-04-15T15:15:43.588533+0000","last_undegraded":"2024-04-15T15:15:43.588533+0000","last_fullsized":"2024-04-15T15:15:43.588533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:03:33.287332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588553+0000","last_change":"2024-04-15T15:15:42.590642+0000","last_active":"2024-04-15T15:15:43.588553+0000","last_peered":"2024-04-15T15:15:43.588553+0000","last_clean":"2024-04-15T15:15:43.588553+0000","last_became_active":"2024-04-15T15:15:42.590216+0000","last_became_peered":"2024-04-15T15:15:42.590216+0000","last_unstale":"2024-04-15T15:15:43.588553+0000","last_undegraded":"2024-04-15T15:15:43.588553+0000","last_fullsized":"2024-04-15T15:15:43.588553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439674+0000","last_change":"2024-04-15T15:15:38.571884+0000","last_active":"2024-04-15T15:15:44.439674+0000","last_peered":"2024-04-15T15:15:44.439674+0000","last_clean":"2024-04-15T15:15:44.439674+0000","last_became_active":"2024-04-15T15:15:38.571574+0000","last_became_peered":"2024-04-15T15:15:38.571574+0000","last_unstale":"2024-04-15T15:15:44.439674+0000","last_undegraded":"2024-04-15T15:15:44.439674+0000","last_fullsized":"2024-04-15T15:15:44.439674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769091+0000","last_change":"2024-04-15T15:15:37.558157+0000","last_active":"2024-04-15T15:15:43.769091+0000","last_peered":"2024-04-15T15:15:43.769091+0000","last_clean":"2024-04-15T15:15:43.769091+0000","last_became_active":"2024-04-15T15:15:37.557622+0000","last_became_peered":"2024-04-15T15:15:37.557622+0000","last_unstale":"2024-04-15T15:15:43.769091+0000","last_undegraded":"2024-04-15T15:15:43.769091+0000","last_fullsized":"2024-04-15T15:15:43.769091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439409+0000","last_change":"2024-04-15T15:15:42.589002+0000","last_active":"2024-04-15T15:15:44.439409+0000","last_peered":"2024-04-15T15:15:44.439409+0000","last_clean":"2024-04-15T15:15:44.439409+0000","last_became_active":"2024-04-15T15:15:42.588573+0000","last_became_peered":"2024-04-15T15:15:42.588573+0000","last_unstale":"2024-04-15T15:15:44.439409+0000","last_undegraded":"2024-04-15T15:15:44.439409+0000","last_fullsized":"2024-04-15T15:15:44.439409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588824+0000","last_change":"2024-04-15T15:15:38.565226+0000","last_active":"2024-04-15T15:15:43.588824+0000","last_peered":"2024-04-15T15:15:43.588824+0000","last_clean":"2024-04-15T15:15:43.588824+0000","last_became_active":"2024-04-15T15:15:38.564938+0000","last_became_peered":"2024-04-15T15:15:38.564938+0000","last_unstale":"2024-04-15T15:15:43.588824+0000","last_undegraded":"2024-04-15T15:15:43.588824+0000","last_fullsized":"2024-04-15T15:15:43.588824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587872+0000","last_change":"2024-04-15T15:15:37.560743+0000","last_active":"2024-04-15T15:15:43.587872+0000","last_peered":"2024-04-15T15:15:43.587872+0000","last_clean":"2024-04-15T15:15:43.587872+0000","last_became_active":"2024-04-15T15:15:37.560439+0000","last_became_peered":"2024-04-15T15:15:37.560439+0000","last_unstale":"2024-04-15T15:15:43.587872+0000","last_undegraded":"2024-04-15T15:15:43.587872+0000","last_fullsized":"2024-04-15T15:15:43.587872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440270+0000","last_change":"2024-04-15T15:15:42.660245+0000","last_active":"2024-04-15T15:15:44.440270+0000","last_peered":"2024-04-15T15:15:44.440270+0000","last_clean":"2024-04-15T15:15:44.440270+0000","last_became_active":"2024-04-15T15:15:42.659882+0000","last_became_peered":"2024-04-15T15:15:42.659882+0000","last_unstale":"2024-04-15T15:15:44.440270+0000","last_undegraded":"2024-04-15T15:15:44.440270+0000","last_fullsized":"2024-04-15T15:15:44.440270+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:16:55.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440094+0000","last_change":"2024-04-15T15:15:38.572105+0000","last_active":"2024-04-15T15:15:44.440094+0000","last_peered":"2024-04-15T15:15:44.440094+0000","last_clean":"2024-04-15T15:15:44.440094+0000","last_became_active":"2024-04-15T15:15:38.571821+0000","last_became_peered":"2024-04-15T15:15:38.571821+0000","last_unstale":"2024-04-15T15:15:44.440094+0000","last_undegraded":"2024-04-15T15:15:44.440094+0000","last_fullsized":"2024-04-15T15:15:44.440094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440332+0000","last_change":"2024-04-15T15:15:37.562506+0000","last_active":"2024-04-15T15:15:44.440332+0000","last_peered":"2024-04-15T15:15:44.440332+0000","last_clean":"2024-04-15T15:15:44.440332+0000","last_became_active":"2024-04-15T15:15:37.562212+0000","last_became_peered":"2024-04-15T15:15:37.562212+0000","last_unstale":"2024-04-15T15:15:44.440332+0000","last_undegraded":"2024-04-15T15:15:44.440332+0000","last_fullsized":"2024-04-15T15:15:44.440332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440199+0000","last_change":"2024-04-15T15:15:42.656037+0000","last_active":"2024-04-15T15:15:44.440199+0000","last_peered":"2024-04-15T15:15:44.440199+0000","last_clean":"2024-04-15T15:15:44.440199+0000","last_became_active":"2024-04-15T15:15:42.655647+0000","last_became_peered":"2024-04-15T15:15:42.655647+0000","last_unstale":"2024-04-15T15:15:44.440199+0000","last_undegraded":"2024-04-15T15:15:44.440199+0000","last_fullsized":"2024-04-15T15:15:44.440199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:53:58.204060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439284+0000","last_change":"2024-04-15T15:15:38.573064+0000","last_active":"2024-04-15T15:15:44.439284+0000","last_peered":"2024-04-15T15:15:44.439284+0000","last_clean":"2024-04-15T15:15:44.439284+0000","last_became_active":"2024-04-15T15:15:38.572766+0000","last_became_peered":"2024-04-15T15:15:38.572766+0000","last_unstale":"2024-04-15T15:15:44.439284+0000","last_undegraded":"2024-04-15T15:15:44.439284+0000","last_fullsized":"2024-04-15T15:15:44.439284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:53:30.455027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439611+0000","last_change":"2024-04-15T15:15:37.565383+0000","last_active":"2024-04-15T15:15:44.439611+0000","last_peered":"2024-04-15T15:15:44.439611+0000","last_clean":"2024-04-15T15:15:44.439611+0000","last_became_active":"2024-04-15T15:15:37.561007+0000","last_became_peered":"2024-04-15T15:15:37.561007+0000","last_unstale":"2024-04-15T15:15:44.439611+0000","last_undegraded":"2024-04-15T15:15:44.439611+0000","last_fullsized":"2024-04-15T15:15:44.439611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440266+0000","last_change":"2024-04-15T15:15:42.657177+0000","last_active":"2024-04-15T15:15:44.440266+0000","last_peered":"2024-04-15T15:15:44.440266+0000","last_clean":"2024-04-15T15:15:44.440266+0000","last_became_active":"2024-04-15T15:15:42.656776+0000","last_became_peered":"2024-04-15T15:15:42.656776+0000","last_unstale":"2024-04-15T15:15:44.440266+0000","last_undegraded":"2024-04-15T15:15:44.440266+0000","last_fullsized":"2024-04-15T15:15:44.440266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439546+0000","last_change":"2024-04-15T15:15:38.569532+0000","last_active":"2024-04-15T15:15:44.439546+0000","last_peered":"2024-04-15T15:15:44.439546+0000","last_clean":"2024-04-15T15:15:44.439546+0000","last_became_active":"2024-04-15T15:15:38.569097+0000","last_became_peered":"2024-04-15T15:15:38.569097+0000","last_unstale":"2024-04-15T15:15:44.439546+0000","last_undegraded":"2024-04-15T15:15:44.439546+0000","last_fullsized":"2024-04-15T15:15:44.439546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:48.573127+0000","last_change":"2024-04-15T15:15:37.559420+0000","last_active":"2024-04-15T15:15:48.573127+0000","last_peered":"2024-04-15T15:15:48.573127+0000","last_clean":"2024-04-15T15:15:48.573127+0000","last_became_active":"2024-04-15T15:15:37.559152+0000","last_became_peered":"2024-04-15T15:15:37.559152+0000","last_unstale":"2024-04-15T15:15:48.573127+0000","last_undegraded":"2024-04-15T15:15:48.573127+0000","last_fullsized":"2024-04-15T15:15:48.573127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587812+0000","last_change":"2024-04-15T15:15:42.591766+0000","last_active":"2024-04-15T15:15:43.587812+0000","last_peered":"2024-04-15T15:15:43.587812+0000","last_clean":"2024-04-15T15:15:43.587812+0000","last_became_active":"2024-04-15T15:15:42.591354+0000","last_became_peered":"2024-04-15T15:15:42.591354+0000","last_unstale":"2024-04-15T15:15:43.587812+0000","last_undegraded":"2024-04-15T15:15:43.587812+0000","last_fullsized":"2024-04-15T15:15:43.587812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588450+0000","last_change":"2024-04-15T15:15:38.562820+0000","last_active":"2024-04-15T15:15:43.588450+0000","last_peered":"2024-04-15T15:15:43.588450+0000","last_clean":"2024-04-15T15:15:43.588450+0000","last_became_active":"2024-04-15T15:15:38.562536+0000","last_became_peered":"2024-04-15T15:15:38.562536+0000","last_unstale":"2024-04-15T15:15:43.588450+0000","last_undegraded":"2024-04-15T15:15:43.588450+0000","last_fullsized":"2024-04-15T15:15:43.588450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587577+0000","last_change":"2024-04-15T15:15:37.558919+0000","last_active":"2024-04-15T15:15:43.587577+0000","last_peered":"2024-04-15T15:15:43.587577+0000","last_clean":"2024-04-15T15:15:43.587577+0000","last_became_active":"2024-04-15T15:15:37.558592+0000","last_became_peered":"2024-04-15T15:15:37.558592+0000","last_unstale":"2024-04-15T15:15:43.587577+0000","last_undegraded":"2024-04-15T15:15:43.587577+0000","last_fullsized":"2024-04-15T15:15:43.587577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587908+0000","last_change":"2024-04-15T15:15:42.591079+0000","last_active":"2024-04-15T15:15:43.587908+0000","last_peered":"2024-04-15T15:15:43.587908+0000","last_clean":"2024-04-15T15:15:43.587908+0000","last_became_active":"2024-04-15T15:15:42.590633+0000","last_became_peered":"2024-04-15T15:15:42.590633+0000","last_unstale":"2024-04-15T15:15:43.587908+0000","last_undegraded":"2024-04-15T15:15:43.587908+0000","last_fullsized":"2024-04-15T15:15:43.587908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:33.636220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439372+0000","last_change":"2024-04-15T15:15:38.574069+0000","last_active":"2024-04-15T15:15:44.439372+0000","last_peered":"2024-04-15T15:15:44.439372+0000","last_clean":"2024-04-15T15:15:44.439372+0000","last_became_active":"2024-04-15T15:15:38.573139+0000","last_became_peered":"2024-04-15T15:15:38.573139+0000","last_unstale":"2024-04-15T15:15:44.439372+0000","last_undegraded":"2024-04-15T15:15:44.439372+0000","last_fullsized":"2024-04-15T15:15:44.439372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440496+0000","last_change":"2024-04-15T15:15:37.563424+0000","last_active":"2024-04-15T15:15:44.440496+0000","last_peered":"2024-04-15T15:15:44.440496+0000","last_clean":"2024-04-15T15:15:44.440496+0000","last_became_active":"2024-04-15T15:15:37.562924+0000","last_became_peered":"2024-04-15T15:15:37.562924+0000","last_unstale":"2024-04-15T15:15:44.440496+0000","last_undegraded":"2024-04-15T15:15:44.440496+0000","last_fullsized":"2024-04-15T15:15:44.440496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439129+0000","last_change":"2024-04-15T15:15:42.587632+0000","last_active":"2024-04-15T15:15:44.439129+0000","last_peered":"2024-04-15T15:15:44.439129+0000","last_clean":"2024-04-15T15:15:44.439129+0000","last_became_active":"2024-04-15T15:15:42.587139+0000","last_became_peered":"2024-04-15T15:15:42.587139+0000","last_unstale":"2024-04-15T15:15:44.439129+0000","last_undegraded":"2024-04-15T15:15:44.439129+0000","last_fullsized":"2024-04-15T15:15:44.439129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440397+0000","last_change":"2024-04-15T15:15:38.567493+0000","last_active":"2024-04-15T15:15:44.440397+0000","last_peered":"2024-04-15T15:15:44.440397+0000","last_clean":"2024-04-15T15:15:44.440397+0000","last_became_active":"2024-04-15T15:15:38.567188+0000","last_became_peered":"2024-04-15T15:15:38.567188+0000","last_unstale":"2024-04-15T15:15:44.440397+0000","last_undegraded":"2024-04-15T15:15:44.440397+0000","last_fullsized":"2024-04-15T15:15:44.440397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588898+0000","last_change":"2024-04-15T15:15:37.563986+0000","last_active":"2024-04-15T15:15:43.588898+0000","last_peered":"2024-04-15T15:15:43.588898+0000","last_clean":"2024-04-15T15:15:43.588898+0000","last_became_active":"2024-04-15T15:15:37.563666+0000","last_became_peered":"2024-04-15T15:15:37.563666+0000","last_unstale":"2024-04-15T15:15:43.588898+0000","last_undegraded":"2024-04-15T15:15:43.588898+0000","last_fullsized":"2024-04-15T15:15:43.588898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440053+0000","last_change":"2024-04-15T15:15:42.661474+0000","last_active":"2024-04-15T15:15:44.440053+0000","last_peered":"2024-04-15T15:15:44.440053+0000","last_clean":"2024-04-15T15:15:44.440053+0000","last_became_active":"2024-04-15T15:15:42.660740+0000","last_became_peered":"2024-04-15T15:15:42.660740+0000","last_unstale":"2024-04-15T15:15:44.440053+0000","last_undegraded":"2024-04-15T15:15:44.440053+0000","last_fullsized":"2024-04-15T15:15:44.440053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:56:48.355956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439422+0000","last_change":"2024-04-15T15:15:42.658445+0000","last_active":"2024-04-15T15:15:44.439422+0000","last_peered":"2024-04-15T15:15:44.439422+0000","last_clean":"2024-04-15T15:15:44.439422+0000","last_became_active":"2024-04-15T15:15:42.658104+0000","last_became_peered":"2024-04-15T15:15:42.658104+0000","last_unstale":"2024-04-15T15:15:44.439422+0000","last_undegraded":"2024-04-15T15:15:44.439422+0000","last_fullsized":"2024-04-15T15:15:44.439422+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586589+0000","last_change":"2024-04-15T15:15:38.562893+0000","last_active":"2024-04-15T15:15:43.586589+0000","last_peered":"2024-04-15T15:15:43.586589+0000","last_clean":"2024-04-15T15:15:43.586589+0000","last_became_active":"2024-04-15T15:15:38.562604+0000","last_became_peered":"2024-04-15T15:15:38.562604+0000","last_unstale":"2024-04-15T15:15:43.586589+0000","last_undegraded":"2024-04-15T15:15:43.586589+0000","last_fullsized":"2024-04-15T15:15:43.586589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:37.556524+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:37.556249+0000","last_became_peered":"2024-04-15T15:15:37.556249+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588777+0000","last_change":"2024-04-15T15:15:42.590110+0000","last_active":"2024-04-15T15:15:43.588777+0000","last_peered":"2024-04-15T15:15:43.588777+0000","last_clean":"2024-04-15T15:15:43.588777+0000","last_became_active":"2024-04-15T15:15:42.589355+0000","last_became_peered":"2024-04-15T15:15:42.589355+0000","last_unstale":"2024-04-15T15:15:43.588777+0000","last_undegraded":"2024-04-15T15:15:43.588777+0000","last_fullsized":"2024-04-15T15:15:43.588777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439180+0000","last_change":"2024-04-15T15:15:38.567772+0000","last_active":"2024-04-15T15:15:44.439180+0000","last_peered":"2024-04-15T15:15:44.439180+0000","last_clean":"2024-04-15T15:15:44.439180+0000","last_became_active":"2024-04-15T15:15:38.566987+0000","last_became_peered":"2024-04-15T15:15:38.566987+0000","last_unstale":"2024-04-15T15:15:44.439180+0000","last_undegraded":"2024-04-15T15:15:44.439180+0000","last_fullsized":"2024-04-15T15:15:44.439180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440394+0000","last_change":"2024-04-15T15:15:37.561996+0000","last_active":"2024-04-15T15:15:44.440394+0000","last_peered":"2024-04-15T15:15:44.440394+0000","last_clean":"2024-04-15T15:15:44.440394+0000","last_became_active":"2024-04-15T15:15:37.561696+0000","last_became_peered":"2024-04-15T15:15:37.561696+0000","last_unstale":"2024-04-15T15:15:44.440394+0000","last_undegraded":"2024-04-15T15:15:44.440394+0000","last_fullsized":"2024-04-15T15:15:44.440394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588032+0000","last_change":"2024-04-15T15:15:37.561778+0000","last_active":"2024-04-15T15:15:43.588032+0000","last_peered":"2024-04-15T15:15:43.588032+0000","last_clean":"2024-04-15T15:15:43.588032+0000","last_became_active":"2024-04-15T15:15:37.561489+0000","last_became_peered":"2024-04-15T15:15:37.561489+0000","last_unstale":"2024-04-15T15:15:43.588032+0000","last_undegraded":"2024-04-15T15:15:43.588032+0000","last_fullsized":"2024-04-15T15:15:43.588032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439557+0000","last_change":"2024-04-15T15:15:38.570137+0000","last_active":"2024-04-15T15:15:44.439557+0000","last_peered":"2024-04-15T15:15:44.439557+0000","last_clean":"2024-04-15T15:15:44.439557+0000","last_became_active":"2024-04-15T15:15:38.569657+0000","last_became_peered":"2024-04-15T15:15:38.569657+0000","last_unstale":"2024-04-15T15:15:44.439557+0000","last_undegraded":"2024-04-15T15:15:44.439557+0000","last_fullsized":"2024-04-15T15:15:44.439557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769366+0000","last_change":"2024-04-15T15:15:42.584680+0000","last_active":"2024-04-15T15:15:43.769366+0000","last_peered":"2024-04-15T15:15:43.769366+0000","last_clean":"2024-04-15T15:15:43.769366+0000","last_became_active":"2024-04-15T15:15:42.584312+0000","last_became_peered":"2024-04-15T15:15:42.584312+0000","last_unstale":"2024-04-15T15:15:43.769366+0000","last_undegraded":"2024-04-15T15:15:43.769366+0000","last_fullsized":"2024-04-15T15:15:43.769366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:39.473475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440046+0000","last_change":"2024-04-15T15:15:37.565397+0000","last_active":"2024-04-15T15:15:44.440046+0000","last_peered":"2024-04-15T15:15:44.440046+0000","last_clean":"2024-04-15T15:15:44.440046+0000","last_became_active":"2024-04-15T15:15:37.565018+0000","last_became_peered":"2024-04-15T15:15:37.565018+0000","last_unstale":"2024-04-15T15:15:44.440046+0000","last_undegraded":"2024-04-15T15:15:44.440046+0000","last_fullsized":"2024-04-15T15:15:44.440046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587469+0000","last_change":"2024-04-15T15:15:38.558591+0000","last_active":"2024-04-15T15:15:43.587469+0000","last_peered":"2024-04-15T15:15:43.587469+0000","last_clean":"2024-04-15T15:15:43.587469+0000","last_became_active":"2024-04-15T15:15:38.558044+0000","last_became_peered":"2024-04-15T15:15:38.558044+0000","last_unstale":"2024-04-15T15:15:43.587469+0000","last_undegraded":"2024-04-15T15:15:43.587469+0000","last_fullsized":"2024-04-15T15:15:43.587469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:56.009007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439916+0000","last_change":"2024-04-15T15:15:42.587509+0000","last_active":"2024-04-15T15:15:44.439916+0000","last_peered":"2024-04-15T15:15:44.439916+0000","last_clean":"2024-04-15T15:15:44.439916+0000","last_became_active":"2024-04-15T15:15:42.587097+0000","last_became_peered":"2024-04-15T15:15:42.587097+0000","last_unstale":"2024-04-15T15:15:44.439916+0000","last_undegraded":"2024-04-15T15:15:44.439916+0000","last_fullsized":"2024-04-15T15:15:44.439916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:55:01.099477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588741+0000","last_change":"2024-04-15T15:15:38.566026+0000","last_active":"2024-04-15T15:15:43.588741+0000","last_peered":"2024-04-15T15:15:43.588741+0000","last_clean":"2024-04-15T15:15:43.588741+0000","last_became_active":"2024-04-15T15:15:38.565726+0000","last_became_peered":"2024-04-15T15:15:38.565726+0000","last_unstale":"2024-04-15T15:15:43.588741+0000","last_undegraded":"2024-04-15T15:15:43.588741+0000","last_fullsized":"2024-04-15T15:15:43.588741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769196+0000","last_change":"2024-04-15T15:15:38.567487+0000","last_active":"2024-04-15T15:15:43.769196+0000","last_peered":"2024-04-15T15:15:43.769196+0000","last_clean":"2024-04-15T15:15:43.769196+0000","last_became_active":"2024-04-15T15:15:38.567201+0000","last_became_peered":"2024-04-15T15:15:38.567201+0000","last_unstale":"2024-04-15T15:15:43.769196+0000","last_undegraded":"2024-04-15T15:15:43.769196+0000","last_fullsized":"2024-04-15T15:15:43.769196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440239+0000","last_change":"2024-04-15T15:15:38.569794+0000","last_active":"2024-04-15T15:15:44.440239+0000","last_peered":"2024-04-15T15:15:44.440239+0000","last_clean":"2024-04-15T15:15:44.440239+0000","last_became_active":"2024-04-15T15:15:38.569510+0000","last_became_peered":"2024-04-15T15:15:38.569510+0000","last_unstale":"2024-04-15T15:15:44.440239+0000","last_undegraded":"2024-04-15T15:15:44.440239+0000","last_fullsized":"2024-04-15T15:15:44.440239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439457+0000","last_change":"2024-04-15T15:15:38.563215+0000","last_active":"2024-04-15T15:15:44.439457+0000","last_peered":"2024-04-15T15:15:44.439457+0000","last_clean":"2024-04-15T15:15:44.439457+0000","last_became_active":"2024-04-15T15:15:38.562942+0000","last_became_peered":"2024-04-15T15:15:38.562942+0000","last_unstale":"2024-04-15T15:15:44.439457+0000","last_undegraded":"2024-04-15T15:15:44.439457+0000","last_fullsized":"2024-04-15T15:15:44.439457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440355+0000","last_change":"2024-04-15T15:15:38.567774+0000","last_active":"2024-04-15T15:15:44.440355+0000","last_peered":"2024-04-15T15:15:44.440355+0000","last_clean":"2024-04-15T15:15:44.440355+0000","last_became_active":"2024-04-15T15:15:38.567471+0000","last_became_peered":"2024-04-15T15:15:38.567471+0000","last_unstale":"2024-04-15T15:15:44.440355+0000","last_undegraded":"2024-04-15T15:15:44.440355+0000","last_fullsized":"2024-04-15T15:15:44.440355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587284+0000","last_change":"2024-04-15T15:15:38.562963+0000","last_active":"2024-04-15T15:15:43.587284+0000","last_peered":"2024-04-15T15:15:43.587284+0000","last_clean":"2024-04-15T15:15:43.587284+0000","last_became_active":"2024-04-15T15:15:38.562611+0000","last_became_peered":"2024-04-15T15:15:38.562611+0000","last_unstale":"2024-04-15T15:15:43.587284+0000","last_undegraded":"2024-04-15T15:15:43.587284+0000","last_fullsized":"2024-04-15T15:15:43.587284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588859+0000","last_change":"2024-04-15T15:15:38.566158+0000","last_active":"2024-04-15T15:15:43.588859+0000","last_peered":"2024-04-15T15:15:43.588859+0000","last_clean":"2024-04-15T15:15:43.588859+0000","last_became_active":"2024-04-15T15:15:38.565859+0000","last_became_peered":"2024-04-15T15:15:38.565859+0000","last_unstale":"2024-04-15T15:15:43.588859+0000","last_undegraded":"2024-04-15T15:15:43.588859+0000","last_fullsized":"2024-04-15T15:15:43.588859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440318+0000","last_change":"2024-04-15T15:15:38.566715+0000","last_active":"2024-04-15T15:15:44.440318+0000","last_peered":"2024-04-15T15:15:44.440318+0000","last_clean":"2024-04-15T15:15:44.440318+0000","last_became_active":"2024-04-15T15:15:38.565677+0000","last_became_peered":"2024-04-15T15:15:38.565677+0000","last_unstale":"2024-04-15T15:15:44.440318+0000","last_undegraded":"2024-04-15T15:15:44.440318+0000","last_fullsized":"2024-04-15T15:15:44.440318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439302+0000","last_change":"2024-04-15T15:15:38.565562+0000","last_active":"2024-04-15T15:15:44.439302+0000","last_peered":"2024-04-15T15:15:44.439302+0000","last_clean":"2024-04-15T15:15:44.439302+0000","last_became_active":"2024-04-15T15:15:38.564779+0000","last_became_peered":"2024-04-15T15:15:38.564779+0000","last_unstale":"2024-04-15T15:15:44.439302+0000","last_undegraded":"2024-04-15T15:15:44.439302+0000","last_fullsized":"2024-04-15T15:15:44.439302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440302+0000","last_change":"2024-04-15T15:15:38.557876+0000","last_active":"2024-04-15T15:15:44.440302+0000","last_peered":"2024-04-15T15:15:44.440302+0000","last_clean":"2024-04-15T15:15:44.440302+0000","last_became_active":"2024-04-15T15:15:38.557568+0000","last_became_peered":"2024-04-15T15:15:38.557568+0000","last_unstale":"2024-04-15T15:15:44.440302+0000","last_undegraded":"2024-04-15T15:15:44.440302+0000","last_fullsized":"2024-04-15T15:15:44.440302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440139+0000","last_change":"2024-04-15T15:15:38.568437+0000","last_active":"2024-04-15T15:15:44.440139+0000","last_peered":"2024-04-15T15:15:44.440139+0000","last_clean":"2024-04-15T15:15:44.440139+0000","last_became_active":"2024-04-15T15:15:38.567071+0000","last_became_peered":"2024-04-15T15:15:38.567071+0000","last_unstale":"2024-04-15T15:15:44.440139+0000","last_undegraded":"2024-04-15T15:15:44.440139+0000","last_fullsized":"2024-04-15T15:15:44.440139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439337+0000","last_change":"2024-04-15T15:15:38.572158+0000","last_active":"2024-04-15T15:15:44.439337+0000","last_peered":"2024-04-15T15:15:44.439337+0000","last_clean":"2024-04-15T15:15:44.439337+0000","last_became_active":"2024-04-15T15:15:38.571868+0000","last_became_peered":"2024-04-15T15:15:38.571868+0000","last_unstale":"2024-04-15T15:15:44.439337+0000","last_undegraded":"2024-04-15T15:15:44.439337+0000","last_fullsized":"2024-04-15T15:15:44.439337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588528+0000","last_change":"2024-04-15T15:15:38.556771+0000","last_active":"2024-04-15T15:15:43.588528+0000","last_peered":"2024-04-15T15:15:43.588528+0000","last_clean":"2024-04-15T15:15:43.588528+0000","last_became_active":"2024-04-15T15:15:38.556456+0000","last_became_peered":"2024-04-15T15:15:38.556456+0000","last_unstale":"2024-04-15T15:15:43.588528+0000","last_undegraded":"2024-04-15T15:15:43.588528+0000","last_fullsized":"2024-04-15T15:15:43.588528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439982+0000","last_change":"2024-04-15T15:15:38.565285+0000","last_active":"2024-04-15T15:15:44.439982+0000","last_peered":"2024-04-15T15:15:44.439982+0000","last_clean":"2024-04-15T15:15:44.439982+0000","last_became_active":"2024-04-15T15:15:38.564505+0000","last_became_peered":"2024-04-15T15:15:38.564505+0000","last_unstale":"2024-04-15T15:15:44.439982+0000","last_undegraded":"2024-04-15T15:15:44.439982+0000","last_fullsized":"2024-04-15T15:15:44.439982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:44.185650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769488+0000","last_change":"2024-04-15T15:15:38.562279+0000","last_active":"2024-04-15T15:15:43.769488+0000","last_peered":"2024-04-15T15:15:43.769488+0000","last_clean":"2024-04-15T15:15:43.769488+0000","last_became_active":"2024-04-15T15:15:38.561971+0000","last_became_peered":"2024-04-15T15:15:38.561971+0000","last_unstale":"2024-04-15T15:15:43.769488+0000","last_undegraded":"2024-04-15T15:15:43.769488+0000","last_fullsized":"2024-04-15T15:15:43.769488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440908+0000","last_change":"2024-04-15T15:15:38.569181+0000","last_active":"2024-04-15T15:15:44.440908+0000","last_peered":"2024-04-15T15:15:44.440908+0000","last_clean":"2024-04-15T15:15:44.440908+0000","last_became_active":"2024-04-15T15:15:38.566602+0000","last_became_peered":"2024-04-15T15:15:38.566602+0000","last_unstale":"2024-04-15T15:15:44.440908+0000","last_undegraded":"2024-04-15T15:15:44.440908+0000","last_fullsized":"2024-04-15T15:15:44.440908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439191+0000","last_change":"2024-04-15T15:15:38.568218+0000","last_active":"2024-04-15T15:15:44.439191+0000","last_peered":"2024-04-15T15:15:44.439191+0000","last_clean":"2024-04-15T15:15:44.439191+0000","last_became_active":"2024-04-15T15:15:38.566966+0000","last_became_peered":"2024-04-15T15:15:38.566966+0000","last_unstale":"2024-04-15T15:15:44.439191+0000","last_undegraded":"2024-04-15T15:15:44.439191+0000","last_fullsized":"2024-04-15T15:15:44.439191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586673+0000","last_change":"2024-04-15T15:15:38.563656+0000","last_active":"2024-04-15T15:15:43.586673+0000","last_peered":"2024-04-15T15:15:43.586673+0000","last_clean":"2024-04-15T15:15:43.586673+0000","last_became_active":"2024-04-15T15:15:38.563380+0000","last_became_peered":"2024-04-15T15:15:38.563380+0000","last_unstale":"2024-04-15T15:15:43.586673+0000","last_undegraded":"2024-04-15T15:15:43.586673+0000","last_fullsized":"2024-04-15T15:15:43.586673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439190+0000","last_change":"2024-04-15T15:15:38.558804+0000","last_active":"2024-04-15T15:15:44.439190+0000","last_peered":"2024-04-15T15:15:44.439190+0000","last_clean":"2024-04-15T15:15:44.439190+0000","last_became_active":"2024-04-15T15:15:38.558374+0000","last_became_peered":"2024-04-15T15:15:38.558374+0000","last_unstale":"2024-04-15T15:15:44.439190+0000","last_undegraded":"2024-04-15T15:15:44.439190+0000","last_fullsized":"2024-04-15T15:15:44.439190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:35.448266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439861+0000","last_change":"2024-04-15T15:15:38.564455+0000","last_active":"2024-04-15T15:15:44.439861+0000","last_peered":"2024-04-15T15:15:44.439861+0000","last_clean":"2024-04-15T15:15:44.439861+0000","last_became_active":"2024-04-15T15:15:38.563328+0000","last_became_peered":"2024-04-15T15:15:38.563328+0000","last_unstale":"2024-04-15T15:15:44.439861+0000","last_undegraded":"2024-04-15T15:15:44.439861+0000","last_fullsized":"2024-04-15T15:15:44.439861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588424+0000","last_change":"2024-04-15T15:15:38.563202+0000","last_active":"2024-04-15T15:15:43.588424+0000","last_peered":"2024-04-15T15:15:43.588424+0000","last_clean":"2024-04-15T15:15:43.588424+0000","last_became_active":"2024-04-15T15:15:38.562862+0000","last_became_peered":"2024-04-15T15:15:38.562862+0000","last_unstale":"2024-04-15T15:15:43.588424+0000","last_undegraded":"2024-04-15T15:15:43.588424+0000","last_fullsized":"2024-04-15T15:15:43.588424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588478+0000","last_change":"2024-04-15T15:15:38.570615+0000","last_active":"2024-04-15T15:15:43.588478+0000","last_peered":"2024-04-15T15:15:43.588478+0000","last_clean":"2024-04-15T15:15:43.588478+0000","last_became_active":"2024-04-15T15:15:38.570340+0000","last_became_peered":"2024-04-15T15:15:38.570340+0000","last_unstale":"2024-04-15T15:15:43.588478+0000","last_undegraded":"2024-04-15T15:15:43.588478+0000","last_fullsized":"2024-04-15T15:15:43.588478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588803+0000","last_change":"2024-04-15T15:15:38.567756+0000","last_active":"2024-04-15T15:15:43.588803+0000","last_peered":"2024-04-15T15:15:43.588803+0000","last_clean":"2024-04-15T15:15:43.588803+0000","last_became_active":"2024-04-15T15:15:38.567455+0000","last_became_peered":"2024-04-15T15:15:38.567455+0000","last_unstale":"2024-04-15T15:15:43.588803+0000","last_undegraded":"2024-04-15T15:15:43.588803+0000","last_fullsized":"2024-04-15T15:15:43.588803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440004+0000","last_change":"2024-04-15T15:15:38.574989+0000","last_active":"2024-04-15T15:15:44.440004+0000","last_peered":"2024-04-15T15:15:44.440004+0000","last_clean":"2024-04-15T15:15:44.440004+0000","last_became_active":"2024-04-15T15:15:38.573776+0000","last_became_peered":"2024-04-15T15:15:38.573776+0000","last_unstale":"2024-04-15T15:15:44.440004+0000","last_undegraded":"2024-04-15T15:15:44.440004+0000","last_fullsized":"2024-04-15T15:15:44.440004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:41.104693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438915+0000","last_change":"2024-04-15T15:15:38.573020+0000","last_active":"2024-04-15T15:15:44.438915+0000","last_peered":"2024-04-15T15:15:44.438915+0000","last_clean":"2024-04-15T15:15:44.438915+0000","last_became_active":"2024-04-15T15:15:38.572715+0000","last_became_peered":"2024-04-15T15:15:38.572715+0000","last_unstale":"2024-04-15T15:15:44.438915+0000","last_undegraded":"2024-04-15T15:15:44.438915+0000","last_fullsized":"2024-04-15T15:15:44.438915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439587+0000","last_change":"2024-04-15T15:15:38.568749+0000","last_active":"2024-04-15T15:15:44.439587+0000","last_peered":"2024-04-15T15:15:44.439587+0000","last_clean":"2024-04-15T15:15:44.439587+0000","last_became_active":"2024-04-15T15:15:38.568488+0000","last_became_peered":"2024-04-15T15:15:38.568488+0000","last_unstale":"2024-04-15T15:15:44.439587+0000","last_undegraded":"2024-04-15T15:15:44.439587+0000","last_fullsized":"2024-04-15T15:15:44.439587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440025+0000","last_change":"2024-04-15T15:15:38.572663+0000","last_active":"2024-04-15T15:15:44.440025+0000","last_peered":"2024-04-15T15:15:44.440025+0000","last_clean":"2024-04-15T15:15:44.440025+0000","last_became_active":"2024-04-15T15:15:38.572349+0000","last_became_peered":"2024-04-15T15:15:38.572349+0000","last_unstale":"2024-04-15T15:15:44.440025+0000","last_undegraded":"2024-04-15T15:15:44.440025+0000","last_fullsized":"2024-04-15T15:15:44.440025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439869+0000","last_change":"2024-04-15T15:15:38.567565+0000","last_active":"2024-04-15T15:15:44.439869+0000","last_peered":"2024-04-15T15:15:44.439869+0000","last_clean":"2024-04-15T15:15:44.439869+0000","last_became_active":"2024-04-15T15:15:38.567300+0000","last_became_peered":"2024-04-15T15:15:38.567300+0000","last_unstale":"2024-04-15T15:15:44.439869+0000","last_undegraded":"2024-04-15T15:15:44.439869+0000","last_fullsized":"2024-04-15T15:15:44.439869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440752+0000","last_change":"2024-04-15T15:15:38.578049+0000","last_active":"2024-04-15T15:15:44.440752+0000","last_peered":"2024-04-15T15:15:44.440752+0000","last_clean":"2024-04-15T15:15:44.440752+0000","last_became_active":"2024-04-15T15:15:38.577801+0000","last_became_peered":"2024-04-15T15:15:38.577801+0000","last_unstale":"2024-04-15T15:15:44.440752+0000","last_undegraded":"2024-04-15T15:15:44.440752+0000","last_fullsized":"2024-04-15T15:15:44.440752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439206+0000","last_change":"2024-04-15T15:15:38.561841+0000","last_active":"2024-04-15T15:15:44.439206+0000","last_peered":"2024-04-15T15:15:44.439206+0000","last_clean":"2024-04-15T15:15:44.439206+0000","last_became_active":"2024-04-15T15:15:38.561554+0000","last_became_peered":"2024-04-15T15:15:38.561554+0000","last_unstale":"2024-04-15T15:15:44.439206+0000","last_undegraded":"2024-04-15T15:15:44.439206+0000","last_fullsized":"2024-04-15T15:15:44.439206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587614+0000","last_change":"2024-04-15T15:15:38.560883+0000","last_active":"2024-04-15T15:15:43.587614+0000","last_peered":"2024-04-15T15:15:43.587614+0000","last_clean":"2024-04-15T15:15:43.587614+0000","last_became_active":"2024-04-15T15:15:38.560562+0000","last_became_peered":"2024-04-15T15:15:38.560562+0000","last_unstale":"2024-04-15T15:15:43.587614+0000","last_undegraded":"2024-04-15T15:15:43.587614+0000","last_fullsized":"2024-04-15T15:15:43.587614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439744+0000","last_change":"2024-04-15T15:15:38.565038+0000","last_active":"2024-04-15T15:15:44.439744+0000","last_peered":"2024-04-15T15:15:44.439744+0000","last_clean":"2024-04-15T15:15:44.439744+0000","last_became_active":"2024-04-15T15:15:38.564551+0000","last_became_peered":"2024-04-15T15:15:38.564551+0000","last_unstale":"2024-04-15T15:15:44.439744+0000","last_undegraded":"2024-04-15T15:15:44.439744+0000","last_fullsized":"2024-04-15T15:15:44.439744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":104213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":695037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":104087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":104247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":868352,"data_stored":695093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:02.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.794+0000 7f1fe4a88640 1 Processor -- start 2024-04-15T15:16:02.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.794+0000 7f1fe4a88640 1 -- start start 2024-04-15T15:16:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fe4a88640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1fe00d15e0 0x7f1fe00d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fe4a88640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe015eb70 con 0x7f1fe015e790 2024-04-15T15:16:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fe4a88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00d3c80 con 0x7f1fe00d4a40 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fe4a88640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe00d3790 con 0x7f1fe00d15e0 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fde59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1fe015e790 0x7f1fe00b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55566/0 (socket says 172.21.15.46:55566) 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fde59c640 1 -- 172.21.15.46:0/527271691 learned_addr learned my addr 172.21.15.46:0/527271691 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fd5d9b640 1 --2- 172.21.15.46:0/527271691 >> v2:172.21.15.78:3300/0 conn(0x7f1fe00d15e0 0x7f1fe00d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270149470 0 0) 0x7f1fe015eb70 con 0x7f1fe015e790 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fcc003520 con 0x7f1fe015e790 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 698117429 0 0) 0x7f1fcc003520 con 0x7f1fe015e790 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 >> v2:172.21.15.78:3300/0 conn(0x7f1fe00d15e0 msgr2=0x7f1fe00d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 --2- 172.21.15.46:0/527271691 >> v2:172.21.15.78:3300/0 conn(0x7f1fe00d15e0 0x7f1fe00d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 >> v1:172.21.15.78:6789/0 conn(0x7f1fe00d4a40 legacy=0x7f1fe00d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe00d3f60 con 0x7f1fe015e790 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f1fd0001690 con 0x7f1fe015e790 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 >> v1:172.21.15.46:6789/0 conn(0x7f1fe015e790 legacy=0x7f1fe00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc004800 0x7f1fcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fcc0052e0 0x7f1fcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00d3790 con 0x7f1fcc0039b0 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1fcc00c1b0 con 0x7f1fcc0052e0 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.795+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1fcc00c3b0 con 0x7f1fcc004800 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fde59c640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fd5d9b640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc004800 0x7f1fcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fded9d640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fcc0052e0 0x7f1fcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fde59c640 1 -- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc004800 msgr2=0x7f1fcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fde59c640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc004800 0x7f1fcc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fde59c640 1 -- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fcc0052e0 msgr2=0x7f1fcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fde59c640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fcc0052e0 0x7f1fcc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fde59c640 1 -- 172.21.15.46:0/527271691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fcc003520 con 0x7f1fcc0039b0 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.796+0000 7f1fded9d640 1 -- 172.21.15.46:0/527271691 reap_dead start 2024-04-15T15:16:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fde59c640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1fe015ed40 tx=0x7f1fcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d42db56eba83f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0007d10 con 0x7f1fcc0039b0 2024-04-15T15:16:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd0030070 con 0x7f1fcc0039b0 2024-04-15T15:16:02.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fddd9b640 1 -- 172.21.15.46:0/527271691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0005d70 con 0x7f1fcc0039b0 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 -- 172.21.15.46:0/527271691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fcc0039b0 msgr2=0x7f1fcc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1fe015ed40 tx=0x7f1fcc005830 comp rx=0 tx=0).stop 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 -- 172.21.15.46:0/527271691 shutdown_connections 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/527271691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fcc0039b0 0x7f1fcc003dd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 -- 172.21.15.46:0/527271691 >> 172.21.15.46:0/527271691 conn(0x7f1fe0067b40 msgr2=0x7f1fe0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 -- 172.21.15.46:0/527271691 shutdown_connections 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.797+0000 7f1fe4a88640 1 -- 172.21.15.46:0/527271691 wait complete. 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.799+0000 7f1fe4a88640 1 Processor -- start 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.799+0000 7f1fe4a88640 1 -- start start 2024-04-15T15:16:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.799+0000 7f1fe4a88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc0039b0 0x7f1fe0158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 0x7f1fe00c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe00c8640 0x7f1fe00cd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00d1a90 con 0x7f1fe00c9b80 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00d45b0 con 0x7f1fe00c8640 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe00d1c10 con 0x7f1fcc0039b0 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 0x7f1fe00c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 0x7f1fe00c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42024/0 (socket says 172.21.15.46:42024) 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 -- 172.21.15.46:0/824420040 learned_addr learned my addr 172.21.15.46:0/824420040 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fde59c640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc0039b0 0x7f1fe0158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fded9d640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe00c8640 0x7f1fe00cd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 -- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc0039b0 msgr2=0x7f1fe0158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc0039b0 0x7f1fe0158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 -- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe00c8640 msgr2=0x7f1fe00cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe00c8640 0x7f1fe00cd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 -- 172.21.15.46:0/824420040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe00c8ae0 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd5d9b640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 0x7f1fe00c8100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8011a70 tx=0x7f1fc8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800fb50 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc800fcf0 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800fea0 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe00c8dc0 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.800+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe00c9380 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.801+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe00689e0 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.802+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1fc8012040 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.802+0000 7f1fd77fe640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1fc006c8f0 0x7f1fc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.802+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1fc808c540 con 0x7f1fe00c9b80 2024-04-15T15:16:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.803+0000 7f1fde59c640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1fc006c8f0 0x7f1fc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.803+0000 7f1fde59c640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1fc006c8f0 0x7f1fc006edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd0039040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.807+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1fc8019030 con 0x7f1fe00c9b80 2024-04-15T15:16:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.987+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1fe015c0b0 con 0x7f1fc006c8f0 2024-04-15T15:16:02.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.995+0000 7f1fd77fe640 1 -- 172.21.15.46:0/824420040 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f1fe015c0b0 con 0x7f1fc006c8f0 2024-04-15T15:16:03.000 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:03.002 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:16:03.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1fc006c8f0 msgr2=0x7f1fc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1fc006c8f0 0x7f1fc006edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd0039040 comp rx=0 tx=0).stop 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 msgr2=0x7f1fe00c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 0x7f1fe00c8100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8011a70 tx=0x7f1fc8011f30 comp rx=0 tx=0).stop 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 shutdown_connections 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1fc006c8f0 0x7f1fc006edb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1fe00c8640 0x7f1fe00cd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fe00c9b80 0x7f1fe00c8100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 --2- 172.21.15.46:0/824420040 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1fcc0039b0 0x7f1fe0158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 >> 172.21.15.46:0/824420040 conn(0x7f1fe0067b40 msgr2=0x7f1fe00cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 shutdown_connections 2024-04-15T15:16:03.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:02.998+0000 7f1fe4a88640 1 -- 172.21.15.46:0/824420040 wait complete. 2024-04-15T15:16:03.011 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-15T15:16:02.358856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231928,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742792,"statfs":{"total":773094113280,"available":772856619008,"internally_reserved":0,"allocated":3350528,"data_stored":2022712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003396"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439492+0000","last_change":"2024-04-15T15:15:37.566709+0000","last_active":"2024-04-15T15:15:44.439492+0000","last_peered":"2024-04-15T15:15:44.439492+0000","last_clean":"2024-04-15T15:15:44.439492+0000","last_became_active":"2024-04-15T15:15:37.566073+0000","last_became_peered":"2024-04-15T15:15:37.566073+0000","last_unstale":"2024-04-15T15:15:44.439492+0000","last_undegraded":"2024-04-15T15:15:44.439492+0000","last_fullsized":"2024-04-15T15:15:44.439492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:40:55.907098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440303+0000","last_change":"2024-04-15T15:15:38.565494+0000","last_active":"2024-04-15T15:15:44.440303+0000","last_peered":"2024-04-15T15:15:44.440303+0000","last_clean":"2024-04-15T15:15:44.440303+0000","last_became_active":"2024-04-15T15:15:38.563911+0000","last_became_peered":"2024-04-15T15:15:38.563911+0000","last_unstale":"2024-04-15T15:15:44.440303+0000","last_undegraded":"2024-04-15T15:15:44.440303+0000","last_fullsized":"2024-04-15T15:15:44.440303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439886+0000","last_change":"2024-04-15T15:15:42.661119+0000","last_active":"2024-04-15T15:15:44.439886+0000","last_peered":"2024-04-15T15:15:44.439886+0000","last_clean":"2024-04-15T15:15:44.439886+0000","last_became_active":"2024-04-15T15:15:42.660440+0000","last_became_peered":"2024-04-15T15:15:42.660440+0000","last_unstale":"2024-04-15T15:15:44.439886+0000","last_undegraded":"2024-04-15T15:15:44.439886+0000","last_fullsized":"2024-04-15T15:15:44.439886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440400+0000","last_change":"2024-04-15T15:15:37.563320+0000","last_active":"2024-04-15T15:15:44.440400+0000","last_peered":"2024-04-15T15:15:44.440400+0000","last_clean":"2024-04-15T15:15:44.440400+0000","last_became_active":"2024-04-15T15:15:37.563058+0000","last_became_peered":"2024-04-15T15:15:37.563058+0000","last_unstale":"2024-04-15T15:15:44.440400+0000","last_undegraded":"2024-04-15T15:15:44.440400+0000","last_fullsized":"2024-04-15T15:15:44.440400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439174+0000","last_change":"2024-04-15T15:15:38.573022+0000","last_active":"2024-04-15T15:15:44.439174+0000","last_peered":"2024-04-15T15:15:44.439174+0000","last_clean":"2024-04-15T15:15:44.439174+0000","last_became_active":"2024-04-15T15:15:38.572726+0000","last_became_peered":"2024-04-15T15:15:38.572726+0000","last_unstale":"2024-04-15T15:15:44.439174+0000","last_undegraded":"2024-04-15T15:15:44.439174+0000","last_fullsized":"2024-04-15T15:15:44.439174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587093+0000","last_change":"2024-04-15T15:15:42.590697+0000","last_active":"2024-04-15T15:15:43.587093+0000","last_peered":"2024-04-15T15:15:43.587093+0000","last_clean":"2024-04-15T15:15:43.587093+0000","last_became_active":"2024-04-15T15:15:42.590272+0000","last_became_peered":"2024-04-15T15:15:42.590272+0000","last_unstale":"2024-04-15T15:15:43.587093+0000","last_undegraded":"2024-04-15T15:15:43.587093+0000","last_fullsized":"2024-04-15T15:15:43.587093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:10.918812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439863+0000","last_change":"2024-04-15T15:15:37.564953+0000","last_active":"2024-04-15T15:15:44.439863+0000","last_peered":"2024-04-15T15:15:44.439863+0000","last_clean":"2024-04-15T15:15:44.439863+0000","last_became_active":"2024-04-15T15:15:37.564653+0000","last_became_peered":"2024-04-15T15:15:37.564653+0000","last_unstale":"2024-04-15T15:15:44.439863+0000","last_undegraded":"2024-04-15T15:15:44.439863+0000","last_fullsized":"2024-04-15T15:15:44.439863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440820+0000","last_change":"2024-04-15T15:15:38.568986+0000","last_active":"2024-04-15T15:15:44.440820+0000","last_peered":"2024-04-15T15:15:44.440820+0000","last_clean":"2024-04-15T15:15:44.440820+0000","last_became_active":"2024-04-15T15:15:38.566559+0000","last_became_peered":"2024-04-15T15:15:38.566559+0000","last_unstale":"2024-04-15T15:15:44.440820+0000","last_undegraded":"2024-04-15T15:15:44.440820+0000","last_fullsized":"2024-04-15T15:15:44.440820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587815+0000","last_change":"2024-04-15T15:15:42.592706+0000","last_active":"2024-04-15T15:15:43.587815+0000","last_peered":"2024-04-15T15:15:43.587815+0000","last_clean":"2024-04-15T15:15:43.587815+0000","last_became_active":"2024-04-15T15:15:42.592134+0000","last_became_peered":"2024-04-15T15:15:42.592134+0000","last_unstale":"2024-04-15T15:15:43.587815+0000","last_undegraded":"2024-04-15T15:15:43.587815+0000","last_fullsized":"2024-04-15T15:15:43.587815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:41.894568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439525+0000","last_change":"2024-04-15T15:15:37.567857+0000","last_active":"2024-04-15T15:15:44.439525+0000","last_peered":"2024-04-15T15:15:44.439525+0000","last_clean":"2024-04-15T15:15:44.439525+0000","last_became_active":"2024-04-15T15:15:37.567560+0000","last_became_peered":"2024-04-15T15:15:37.567560+0000","last_unstale":"2024-04-15T15:15:44.439525+0000","last_undegraded":"2024-04-15T15:15:44.439525+0000","last_fullsized":"2024-04-15T15:15:44.439525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:36:21.001555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440328+0000","last_change":"2024-04-15T15:15:38.568788+0000","last_active":"2024-04-15T15:15:44.440328+0000","last_peered":"2024-04-15T15:15:44.440328+0000","last_clean":"2024-04-15T15:15:44.440328+0000","last_became_active":"2024-04-15T15:15:38.567501+0000","last_became_peered":"2024-04-15T15:15:38.567501+0000","last_unstale":"2024-04-15T15:15:44.440328+0000","last_undegraded":"2024-04-15T15:15:44.440328+0000","last_fullsized":"2024-04-15T15:15:44.440328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587164+0000","last_change":"2024-04-15T15:15:42.587928+0000","last_active":"2024-04-15T15:15:43.587164+0000","last_peered":"2024-04-15T15:15:43.587164+0000","last_clean":"2024-04-15T15:15:43.587164+0000","last_became_active":"2024-04-15T15:15:42.587518+0000","last_became_peered":"2024-04-15T15:15:42.587518+0000","last_unstale":"2024-04-15T15:15:43.587164+0000","last_undegraded":"2024-04-15T15:15:43.587164+0000","last_fullsized":"2024-04-15T15:15:43.587164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:50.908590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587148+0000","last_change":"2024-04-15T15:15:38.560568+0000","last_active":"2024-04-15T15:15:43.587148+0000","last_peered":"2024-04-15T15:15:43.587148+0000","last_clean":"2024-04-15T15:15:43.587148+0000","last_became_active":"2024-04-15T15:15:38.560292+0000","last_became_peered":"2024-04-15T15:15:38.560292+0000","last_unstale":"2024-04-15T15:15:43.587148+0000","last_undegraded":"2024-04-15T15:15:43.587148+0000","last_fullsized":"2024-04-15T15:15:43.587148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440210+0000","last_change":"2024-04-15T15:15:37.570166+0000","last_active":"2024-04-15T15:15:44.440210+0000","last_peered":"2024-04-15T15:15:44.440210+0000","last_clean":"2024-04-15T15:15:44.440210+0000","last_became_active":"2024-04-15T15:15:37.569862+0000","last_became_peered":"2024-04-15T15:15:37.569862+0000","last_unstale":"2024-04-15T15:15:44.440210+0000","last_undegraded":"2024-04-15T15:15:44.440210+0000","last_fullsized":"2024-04-15T15:15:44.440210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:55:13.182192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588794+0000","last_change":"2024-04-15T15:15:42.590929+0000","last_active":"2024-04-15T15:15:43.588794+0000","last_peered":"2024-04-15T15:15:43.588794+0000","last_clean":"2024-04-15T15:15:43.588794+0000","last_became_active":"2024-04-15T15:15:42.589759+0000","last_became_peered":"2024-04-15T15:15:42.589759+0000","last_unstale":"2024-04-15T15:15:43.588794+0000","last_undegraded":"2024-04-15T15:15:43.588794+0000","last_fullsized":"2024-04-15T15:15:43.588794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439955+0000","last_change":"2024-04-15T15:15:38.563606+0000","last_active":"2024-04-15T15:15:44.439955+0000","last_peered":"2024-04-15T15:15:44.439955+0000","last_clean":"2024-04-15T15:15:44.439955+0000","last_became_active":"2024-04-15T15:15:38.562885+0000","last_became_peered":"2024-04-15T15:15:38.562885+0000","last_unstale":"2024-04-15T15:15:44.439955+0000","last_undegraded":"2024-04-15T15:15:44.439955+0000","last_fullsized":"2024-04-15T15:15:44.439955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587327+0000","last_change":"2024-04-15T15:15:37.562063+0000","last_active":"2024-04-15T15:15:43.587327+0000","last_peered":"2024-04-15T15:15:43.587327+0000","last_clean":"2024-04-15T15:15:43.587327+0000","last_became_active":"2024-04-15T15:15:37.561757+0000","last_became_peered":"2024-04-15T15:15:37.561757+0000","last_unstale":"2024-04-15T15:15:43.587327+0000","last_undegraded":"2024-04-15T15:15:43.587327+0000","last_fullsized":"2024-04-15T15:15:43.587327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588354+0000","last_change":"2024-04-15T15:15:42.591439+0000","last_active":"2024-04-15T15:15:43.588354+0000","last_peered":"2024-04-15T15:15:43.588354+0000","last_clean":"2024-04-15T15:15:43.588354+0000","last_became_active":"2024-04-15T15:15:42.591030+0000","last_became_peered":"2024-04-15T15:15:42.591030+0000","last_unstale":"2024-04-15T15:15:43.588354+0000","last_undegraded":"2024-04-15T15:15:43.588354+0000","last_fullsized":"2024-04-15T15:15:43.588354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:19.368841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439666+0000","last_change":"2024-04-15T15:15:38.566770+0000","last_active":"2024-04-15T15:15:44.439666+0000","last_peered":"2024-04-15T15:15:44.439666+0000","last_clean":"2024-04-15T15:15:44.439666+0000","last_became_active":"2024-04-15T15:15:38.566484+0000","last_became_peered":"2024-04-15T15:15:38.566484+0000","last_unstale":"2024-04-15T15:15:44.439666+0000","last_undegraded":"2024-04-15T15:15:44.439666+0000","last_fullsized":"2024-04-15T15:15:44.439666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586733+0000","last_change":"2024-04-15T15:15:37.559565+0000","last_active":"2024-04-15T15:15:43.586733+0000","last_peered":"2024-04-15T15:15:43.586733+0000","last_clean":"2024-04-15T15:15:43.586733+0000","last_became_active":"2024-04-15T15:15:37.558121+0000","last_became_peered":"2024-04-15T15:15:37.558121+0000","last_unstale":"2024-04-15T15:15:43.586733+0000","last_undegraded":"2024-04-15T15:15:43.586733+0000","last_fullsized":"2024-04-15T15:15:43.586733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.655888+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.655425+0000","last_became_peered":"2024-04-15T15:15:42.655425+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587565+0000","last_change":"2024-04-15T15:15:38.558135+0000","last_active":"2024-04-15T15:15:43.587565+0000","last_peered":"2024-04-15T15:15:43.587565+0000","last_clean":"2024-04-15T15:15:43.587565+0000","last_became_active":"2024-04-15T15:15:38.557849+0000","last_became_peered":"2024-04-15T15:15:38.557849+0000","last_unstale":"2024-04-15T15:15:43.587565+0000","last_undegraded":"2024-04-15T15:15:43.587565+0000","last_fullsized":"2024-04-15T15:15:43.587565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:08.649907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439485+0000","last_change":"2024-04-15T15:15:37.568252+0000","last_active":"2024-04-15T15:15:44.439485+0000","last_peered":"2024-04-15T15:15:44.439485+0000","last_clean":"2024-04-15T15:15:44.439485+0000","last_became_active":"2024-04-15T15:15:37.567982+0000","last_became_peered":"2024-04-15T15:15:37.567982+0000","last_unstale":"2024-04-15T15:15:44.439485+0000","last_undegraded":"2024-04-15T15:15:44.439485+0000","last_fullsized":"2024-04-15T15:15:44.439485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588452+0000","last_change":"2024-04-15T15:15:42.593489+0000","last_active":"2024-04-15T15:15:43.588452+0000","last_peered":"2024-04-15T15:15:43.588452+0000","last_clean":"2024-04-15T15:15:43.588452+0000","last_became_active":"2024-04-15T15:15:42.593152+0000","last_became_peered":"2024-04-15T15:15:42.593152+0000","last_unstale":"2024-04-15T15:15:43.588452+0000","last_undegraded":"2024-04-15T15:15:43.588452+0000","last_fullsized":"2024-04-15T15:15:43.588452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439610+0000","last_change":"2024-04-15T15:15:38.559585+0000","last_active":"2024-04-15T15:15:44.439610+0000","last_peered":"2024-04-15T15:15:44.439610+0000","last_clean":"2024-04-15T15:15:44.439610+0000","last_became_active":"2024-04-15T15:15:38.559225+0000","last_became_peered":"2024-04-15T15:15:38.559225+0000","last_unstale":"2024-04-15T15:15:44.439610+0000","last_undegraded":"2024-04-15T15:15:44.439610+0000","last_fullsized":"2024-04-15T15:15:44.439610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.589097+0000","last_change":"2024-04-15T15:15:37.558650+0000","last_active":"2024-04-15T15:15:43.589097+0000","last_peered":"2024-04-15T15:15:43.589097+0000","last_clean":"2024-04-15T15:15:43.589097+0000","last_became_active":"2024-04-15T15:15:37.558128+0000","last_became_peered":"2024-04-15T15:15:37.558128+0000","last_unstale":"2024-04-15T15:15:43.589097+0000","last_undegraded":"2024-04-15T15:15:43.589097+0000","last_fullsized":"2024-04-15T15:15:43.589097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587388+0000","last_change":"2024-04-15T15:15:42.655806+0000","last_active":"2024-04-15T15:15:43.587388+0000","last_peered":"2024-04-15T15:15:43.587388+0000","last_clean":"2024-04-15T15:15:43.587388+0000","last_became_active":"2024-04-15T15:15:42.655431+0000","last_became_peered":"2024-04-15T15:15:42.655431+0000","last_unstale":"2024-04-15T15:15:43.587388+0000","last_undegraded":"2024-04-15T15:15:43.587388+0000","last_fullsized":"2024-04-15T15:15:43.587388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:10.032080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440420+0000","last_change":"2024-04-15T15:15:38.567366+0000","last_active":"2024-04-15T15:15:44.440420+0000","last_peered":"2024-04-15T15:15:44.440420+0000","last_clean":"2024-04-15T15:15:44.440420+0000","last_became_active":"2024-04-15T15:15:38.567064+0000","last_became_peered":"2024-04-15T15:15:38.567064+0000","last_unstale":"2024-04-15T15:15:44.440420+0000","last_undegraded":"2024-04-15T15:15:44.440420+0000","last_fullsized":"2024-04-15T15:15:44.440420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587807+0000","last_change":"2024-04-15T15:15:37.554351+0000","last_active":"2024-04-15T15:15:43.587807+0000","last_peered":"2024-04-15T15:15:43.587807+0000","last_clean":"2024-04-15T15:15:43.587807+0000","last_became_active":"2024-04-15T15:15:37.554062+0000","last_became_peered":"2024-04-15T15:15:37.554062+0000","last_unstale":"2024-04-15T15:15:43.587807+0000","last_undegraded":"2024-04-15T15:15:43.587807+0000","last_fullsized":"2024-04-15T15:15:43.587807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438909+0000","last_change":"2024-04-15T15:15:42.659938+0000","last_active":"2024-04-15T15:15:44.438909+0000","last_peered":"2024-04-15T15:15:44.438909+0000","last_clean":"2024-04-15T15:15:44.438909+0000","last_became_active":"2024-04-15T15:15:42.659252+0000","last_became_peered":"2024-04-15T15:15:42.659252+0000","last_unstale":"2024-04-15T15:15:44.438909+0000","last_undegraded":"2024-04-15T15:15:44.438909+0000","last_fullsized":"2024-04-15T15:15:44.438909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:48.210859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587105+0000","last_change":"2024-04-15T15:15:38.561075+0000","last_active":"2024-04-15T15:15:43.587105+0000","last_peered":"2024-04-15T15:15:43.587105+0000","last_clean":"2024-04-15T15:15:43.587105+0000","last_became_active":"2024-04-15T15:15:38.559163+0000","last_became_peered":"2024-04-15T15:15:38.559163+0000","last_unstale":"2024-04-15T15:15:43.587105+0000","last_undegraded":"2024-04-15T15:15:43.587105+0000","last_fullsized":"2024-04-15T15:15:43.587105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440117+0000","last_change":"2024-04-15T15:15:37.568261+0000","last_active":"2024-04-15T15:15:44.440117+0000","last_peered":"2024-04-15T15:15:44.440117+0000","last_clean":"2024-04-15T15:15:44.440117+0000","last_became_active":"2024-04-15T15:15:37.567997+0000","last_became_peered":"2024-04-15T15:15:37.567997+0000","last_unstale":"2024-04-15T15:15:44.440117+0000","last_undegraded":"2024-04-15T15:15:44.440117+0000","last_fullsized":"2024-04-15T15:15:44.440117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:47:24.543446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:42.658160+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:42.657766+0000","last_became_peered":"2024-04-15T15:15:42.657766+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440434+0000","last_change":"2024-04-15T15:15:38.563836+0000","last_active":"2024-04-15T15:15:44.440434+0000","last_peered":"2024-04-15T15:15:44.440434+0000","last_clean":"2024-04-15T15:15:44.440434+0000","last_became_active":"2024-04-15T15:15:38.563487+0000","last_became_peered":"2024-04-15T15:15:38.563487+0000","last_unstale":"2024-04-15T15:15:44.440434+0000","last_undegraded":"2024-04-15T15:15:44.440434+0000","last_fullsized":"2024-04-15T15:15:44.440434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587767+0000","last_change":"2024-04-15T15:15:37.563786+0000","last_active":"2024-04-15T15:15:43.587767+0000","last_peered":"2024-04-15T15:15:43.587767+0000","last_clean":"2024-04-15T15:15:43.587767+0000","last_became_active":"2024-04-15T15:15:37.563491+0000","last_became_peered":"2024-04-15T15:15:37.563491+0000","last_unstale":"2024-04-15T15:15:43.587767+0000","last_undegraded":"2024-04-15T15:15:43.587767+0000","last_fullsized":"2024-04-15T15:15:43.587767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586975+0000","last_change":"2024-04-15T15:15:42.593439+0000","last_active":"2024-04-15T15:15:43.586975+0000","last_peered":"2024-04-15T15:15:43.586975+0000","last_clean":"2024-04-15T15:15:43.586975+0000","last_became_active":"2024-04-15T15:15:42.591555+0000","last_became_peered":"2024-04-15T15:15:42.591555+0000","last_unstale":"2024-04-15T15:15:43.586975+0000","last_undegraded":"2024-04-15T15:15:43.586975+0000","last_fullsized":"2024-04-15T15:15:43.586975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588422+0000","last_change":"2024-04-15T15:15:38.560053+0000","last_active":"2024-04-15T15:15:43.588422+0000","last_peered":"2024-04-15T15:15:43.588422+0000","last_clean":"2024-04-15T15:15:43.588422+0000","last_became_active":"2024-04-15T15:15:38.559462+0000","last_became_peered":"2024-04-15T15:15:38.559462+0000","last_unstale":"2024-04-15T15:15:43.588422+0000","last_undegraded":"2024-04-15T15:15:43.588422+0000","last_fullsized":"2024-04-15T15:15:43.588422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:50.427117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440424+0000","last_change":"2024-04-15T15:15:37.561463+0000","last_active":"2024-04-15T15:15:44.440424+0000","last_peered":"2024-04-15T15:15:44.440424+0000","last_clean":"2024-04-15T15:15:44.440424+0000","last_became_active":"2024-04-15T15:15:37.560849+0000","last_became_peered":"2024-04-15T15:15:37.560849+0000","last_unstale":"2024-04-15T15:15:44.440424+0000","last_undegraded":"2024-04-15T15:15:44.440424+0000","last_fullsized":"2024-04-15T15:15:44.440424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.594805+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.594322+0000","last_became_peered":"2024-04-15T15:15:42.594322+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586925+0000","last_change":"2024-04-15T15:15:38.559721+0000","last_active":"2024-04-15T15:15:43.586925+0000","last_peered":"2024-04-15T15:15:43.586925+0000","last_clean":"2024-04-15T15:15:43.586925+0000","last_became_active":"2024-04-15T15:15:38.559440+0000","last_became_peered":"2024-04-15T15:15:38.559440+0000","last_unstale":"2024-04-15T15:15:43.586925+0000","last_undegraded":"2024-04-15T15:15:43.586925+0000","last_fullsized":"2024-04-15T15:15:43.586925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769205+0000","last_change":"2024-04-15T15:15:37.562651+0000","last_active":"2024-04-15T15:15:43.769205+0000","last_peered":"2024-04-15T15:15:43.769205+0000","last_clean":"2024-04-15T15:15:43.769205+0000","last_became_active":"2024-04-15T15:15:37.562267+0000","last_became_peered":"2024-04-15T15:15:37.562267+0000","last_unstale":"2024-04-15T15:15:43.769205+0000","last_undegraded":"2024-04-15T15:15:43.769205+0000","last_fullsized":"2024-04-15T15:15:43.769205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587782+0000","last_change":"2024-04-15T15:15:42.595294+0000","last_active":"2024-04-15T15:15:43.587782+0000","last_peered":"2024-04-15T15:15:43.587782+0000","last_clean":"2024-04-15T15:15:43.587782+0000","last_became_active":"2024-04-15T15:15:42.594395+0000","last_became_peered":"2024-04-15T15:15:42.594395+0000","last_unstale":"2024-04-15T15:15:43.587782+0000","last_undegraded":"2024-04-15T15:15:43.587782+0000","last_fullsized":"2024-04-15T15:15:43.587782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438893+0000","last_change":"2024-04-15T15:15:38.566568+0000","last_active":"2024-04-15T15:15:44.438893+0000","last_peered":"2024-04-15T15:15:44.438893+0000","last_clean":"2024-04-15T15:15:44.438893+0000","last_became_active":"2024-04-15T15:15:38.566261+0000","last_became_peered":"2024-04-15T15:15:38.566261+0000","last_unstale":"2024-04-15T15:15:44.438893+0000","last_undegraded":"2024-04-15T15:15:44.438893+0000","last_fullsized":"2024-04-15T15:15:44.438893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439388+0000","last_change":"2024-04-15T15:15:37.567337+0000","last_active":"2024-04-15T15:15:44.439388+0000","last_peered":"2024-04-15T15:15:44.439388+0000","last_clean":"2024-04-15T15:15:44.439388+0000","last_became_active":"2024-04-15T15:15:37.567042+0000","last_became_peered":"2024-04-15T15:15:37.567042+0000","last_unstale":"2024-04-15T15:15:44.439388+0000","last_undegraded":"2024-04-15T15:15:44.439388+0000","last_fullsized":"2024-04-15T15:15:44.439388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439562+0000","last_change":"2024-04-15T15:15:42.657091+0000","last_active":"2024-04-15T15:15:44.439562+0000","last_peered":"2024-04-15T15:15:44.439562+0000","last_clean":"2024-04-15T15:15:44.439562+0000","last_became_active":"2024-04-15T15:15:42.656638+0000","last_became_peered":"2024-04-15T15:15:42.656638+0000","last_unstale":"2024-04-15T15:15:44.439562+0000","last_undegraded":"2024-04-15T15:15:44.439562+0000","last_fullsized":"2024-04-15T15:15:44.439562+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:46.336878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439991+0000","last_change":"2024-04-15T15:15:38.576736+0000","last_active":"2024-04-15T15:15:44.439991+0000","last_peered":"2024-04-15T15:15:44.439991+0000","last_clean":"2024-04-15T15:15:44.439991+0000","last_became_active":"2024-04-15T15:15:38.574326+0000","last_became_peered":"2024-04-15T15:15:38.574326+0000","last_unstale":"2024-04-15T15:15:44.439991+0000","last_undegraded":"2024-04-15T15:15:44.439991+0000","last_fullsized":"2024-04-15T15:15:44.439991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440769+0000","last_change":"2024-04-15T15:15:37.557351+0000","last_active":"2024-04-15T15:15:44.440769+0000","last_peered":"2024-04-15T15:15:44.440769+0000","last_clean":"2024-04-15T15:15:44.440769+0000","last_became_active":"2024-04-15T15:15:37.557036+0000","last_became_peered":"2024-04-15T15:15:37.557036+0000","last_unstale":"2024-04-15T15:15:44.440769+0000","last_undegraded":"2024-04-15T15:15:44.440769+0000","last_fullsized":"2024-04-15T15:15:44.440769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439564+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:15:44.439564+0000","last_peered":"2024-04-15T15:15:44.439564+0000","last_clean":"2024-04-15T15:15:44.439564+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:15:44.439564+0000","last_undegraded":"2024-04-15T15:15:44.439564+0000","last_fullsized":"2024-04-15T15:15:44.439564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588337+0000","last_change":"2024-04-15T15:15:42.658428+0000","last_active":"2024-04-15T15:15:43.588337+0000","last_peered":"2024-04-15T15:15:43.588337+0000","last_clean":"2024-04-15T15:15:43.588337+0000","last_became_active":"2024-04-15T15:15:42.658014+0000","last_became_peered":"2024-04-15T15:15:42.658014+0000","last_unstale":"2024-04-15T15:15:43.588337+0000","last_undegraded":"2024-04-15T15:15:43.588337+0000","last_fullsized":"2024-04-15T15:15:43.588337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439749+0000","last_change":"2024-04-15T15:15:38.572859+0000","last_active":"2024-04-15T15:15:44.439749+0000","last_peered":"2024-04-15T15:15:44.439749+0000","last_clean":"2024-04-15T15:15:44.439749+0000","last_became_active":"2024-04-15T15:15:38.572580+0000","last_became_peered":"2024-04-15T15:15:38.572580+0000","last_unstale":"2024-04-15T15:15:44.439749+0000","last_undegraded":"2024-04-15T15:15:44.439749+0000","last_fullsized":"2024-04-15T15:15:44.439749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769429+0000","last_change":"2024-04-15T15:15:37.555795+0000","last_active":"2024-04-15T15:15:43.769429+0000","last_peered":"2024-04-15T15:15:43.769429+0000","last_clean":"2024-04-15T15:15:43.769429+0000","last_became_active":"2024-04-15T15:15:37.555502+0000","last_became_peered":"2024-04-15T15:15:37.555502+0000","last_unstale":"2024-04-15T15:15:43.769429+0000","last_undegraded":"2024-04-15T15:15:43.769429+0000","last_fullsized":"2024-04-15T15:15:43.769429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587412+0000","last_change":"2024-04-15T15:15:42.655842+0000","last_active":"2024-04-15T15:15:43.587412+0000","last_peered":"2024-04-15T15:15:43.587412+0000","last_clean":"2024-04-15T15:15:43.587412+0000","last_became_active":"2024-04-15T15:15:42.655470+0000","last_became_peered":"2024-04-15T15:15:42.655470+0000","last_unstale":"2024-04-15T15:15:43.587412+0000","last_undegraded":"2024-04-15T15:15:43.587412+0000","last_fullsized":"2024-04-15T15:15:43.587412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440874+0000","last_change":"2024-04-15T15:15:38.573853+0000","last_active":"2024-04-15T15:15:44.440874+0000","last_peered":"2024-04-15T15:15:44.440874+0000","last_clean":"2024-04-15T15:15:44.440874+0000","last_became_active":"2024-04-15T15:15:38.573585+0000","last_became_peered":"2024-04-15T15:15:38.573585+0000","last_unstale":"2024-04-15T15:15:44.440874+0000","last_undegraded":"2024-04-15T15:15:44.440874+0000","last_fullsized":"2024-04-15T15:15:44.440874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587697+0000","last_change":"2024-04-15T15:15:37.557027+0000","last_active":"2024-04-15T15:15:43.587697+0000","last_peered":"2024-04-15T15:15:43.587697+0000","last_clean":"2024-04-15T15:15:43.587697+0000","last_became_active":"2024-04-15T15:15:37.556725+0000","last_became_peered":"2024-04-15T15:15:37.556725+0000","last_unstale":"2024-04-15T15:15:43.587697+0000","last_undegraded":"2024-04-15T15:15:43.587697+0000","last_fullsized":"2024-04-15T15:15:43.587697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439555+0000","last_change":"2024-04-15T15:15:42.593049+0000","last_active":"2024-04-15T15:15:44.439555+0000","last_peered":"2024-04-15T15:15:44.439555+0000","last_clean":"2024-04-15T15:15:44.439555+0000","last_became_active":"2024-04-15T15:15:42.592494+0000","last_became_peered":"2024-04-15T15:15:42.592494+0000","last_unstale":"2024-04-15T15:15:44.439555+0000","last_undegraded":"2024-04-15T15:15:44.439555+0000","last_fullsized":"2024-04-15T15:15:44.439555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440121+0000","last_change":"2024-04-15T15:15:38.574063+0000","last_active":"2024-04-15T15:15:44.440121+0000","last_peered":"2024-04-15T15:15:44.440121+0000","last_clean":"2024-04-15T15:15:44.440121+0000","last_became_active":"2024-04-15T15:15:38.572802+0000","last_became_peered":"2024-04-15T15:15:38.572802+0000","last_unstale":"2024-04-15T15:15:44.440121+0000","last_undegraded":"2024-04-15T15:15:44.440121+0000","last_fullsized":"2024-04-15T15:15:44.440121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:42.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769497+0000","last_change":"2024-04-15T15:15:37.563114+0000","last_active":"2024-04-15T15:15:43.769497+0000","last_peered":"2024-04-15T15:15:43.769497+0000","last_clean":"2024-04-15T15:15:43.769497+0000","last_became_active":"2024-04-15T15:15:37.562746+0000","last_became_peered":"2024-04-15T15:15:37.562746+0000","last_unstale":"2024-04-15T15:15:43.769497+0000","last_undegraded":"2024-04-15T15:15:43.769497+0000","last_fullsized":"2024-04-15T15:15:43.769497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439239+0000","last_change":"2024-04-15T15:15:42.658786+0000","last_active":"2024-04-15T15:15:44.439239+0000","last_peered":"2024-04-15T15:15:44.439239+0000","last_clean":"2024-04-15T15:15:44.439239+0000","last_became_active":"2024-04-15T15:15:42.658329+0000","last_became_peered":"2024-04-15T15:15:42.658329+0000","last_unstale":"2024-04-15T15:15:44.439239+0000","last_undegraded":"2024-04-15T15:15:44.439239+0000","last_fullsized":"2024-04-15T15:15:44.439239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439930+0000","last_change":"2024-04-15T15:15:38.573032+0000","last_active":"2024-04-15T15:15:44.439930+0000","last_peered":"2024-04-15T15:15:44.439930+0000","last_clean":"2024-04-15T15:15:44.439930+0000","last_became_active":"2024-04-15T15:15:38.572689+0000","last_became_peered":"2024-04-15T15:15:38.572689+0000","last_unstale":"2024-04-15T15:15:44.439930+0000","last_undegraded":"2024-04-15T15:15:44.439930+0000","last_fullsized":"2024-04-15T15:15:44.439930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588533+0000","last_change":"2024-04-15T15:15:37.565326+0000","last_active":"2024-04-15T15:15:43.588533+0000","last_peered":"2024-04-15T15:15:43.588533+0000","last_clean":"2024-04-15T15:15:43.588533+0000","last_became_active":"2024-04-15T15:15:37.564993+0000","last_became_peered":"2024-04-15T15:15:37.564993+0000","last_unstale":"2024-04-15T15:15:43.588533+0000","last_undegraded":"2024-04-15T15:15:43.588533+0000","last_fullsized":"2024-04-15T15:15:43.588533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:03:33.287332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588553+0000","last_change":"2024-04-15T15:15:42.590642+0000","last_active":"2024-04-15T15:15:43.588553+0000","last_peered":"2024-04-15T15:15:43.588553+0000","last_clean":"2024-04-15T15:15:43.588553+0000","last_became_active":"2024-04-15T15:15:42.590216+0000","last_became_peered":"2024-04-15T15:15:42.590216+0000","last_unstale":"2024-04-15T15:15:43.588553+0000","last_undegraded":"2024-04-15T15:15:43.588553+0000","last_fullsized":"2024-04-15T15:15:43.588553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439674+0000","last_change":"2024-04-15T15:15:38.571884+0000","last_active":"2024-04-15T15:15:44.439674+0000","last_peered":"2024-04-15T15:15:44.439674+0000","last_clean":"2024-04-15T15:15:44.439674+0000","last_became_active":"2024-04-15T15:15:38.571574+0000","last_became_peered":"2024-04-15T15:15:38.571574+0000","last_unstale":"2024-04-15T15:15:44.439674+0000","last_undegraded":"2024-04-15T15:15:44.439674+0000","last_fullsized":"2024-04-15T15:15:44.439674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769091+0000","last_change":"2024-04-15T15:15:37.558157+0000","last_active":"2024-04-15T15:15:43.769091+0000","last_peered":"2024-04-15T15:15:43.769091+0000","last_clean":"2024-04-15T15:15:43.769091+0000","last_became_active":"2024-04-15T15:15:37.557622+0000","last_became_peered":"2024-04-15T15:15:37.557622+0000","last_unstale":"2024-04-15T15:15:43.769091+0000","last_undegraded":"2024-04-15T15:15:43.769091+0000","last_fullsized":"2024-04-15T15:15:43.769091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439409+0000","last_change":"2024-04-15T15:15:42.589002+0000","last_active":"2024-04-15T15:15:44.439409+0000","last_peered":"2024-04-15T15:15:44.439409+0000","last_clean":"2024-04-15T15:15:44.439409+0000","last_became_active":"2024-04-15T15:15:42.588573+0000","last_became_peered":"2024-04-15T15:15:42.588573+0000","last_unstale":"2024-04-15T15:15:44.439409+0000","last_undegraded":"2024-04-15T15:15:44.439409+0000","last_fullsized":"2024-04-15T15:15:44.439409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588824+0000","last_change":"2024-04-15T15:15:38.565226+0000","last_active":"2024-04-15T15:15:43.588824+0000","last_peered":"2024-04-15T15:15:43.588824+0000","last_clean":"2024-04-15T15:15:43.588824+0000","last_became_active":"2024-04-15T15:15:38.564938+0000","last_became_peered":"2024-04-15T15:15:38.564938+0000","last_unstale":"2024-04-15T15:15:43.588824+0000","last_undegraded":"2024-04-15T15:15:43.588824+0000","last_fullsized":"2024-04-15T15:15:43.588824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587872+0000","last_change":"2024-04-15T15:15:37.560743+0000","last_active":"2024-04-15T15:15:43.587872+0000","last_peered":"2024-04-15T15:15:43.587872+0000","last_clean":"2024-04-15T15:15:43.587872+0000","last_became_active":"2024-04-15T15:15:37.560439+0000","last_became_peered":"2024-04-15T15:15:37.560439+0000","last_unstale":"2024-04-15T15:15:43.587872+0000","last_undegraded":"2024-04-15T15:15:43.587872+0000","last_fullsized":"2024-04-15T15:15:43.587872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440270+0000","last_change":"2024-04-15T15:15:42.660245+0000","last_active":"2024-04-15T15:15:44.440270+0000","last_peered":"2024-04-15T15:15:44.440270+0000","last_clean":"2024-04-15T15:15:44.440270+0000","last_became_active":"2024-04-15T15:15:42.659882+0000","last_became_peered":"2024-04-15T15:15:42.659882+0000","last_unstale":"2024-04-15T15:15:44.440270+0000","last_undegraded":"2024-04-15T15:15:44.440270+0000","last_fullsized":"2024-04-15T15:15:44.440270+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:16:55.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440094+0000","last_change":"2024-04-15T15:15:38.572105+0000","last_active":"2024-04-15T15:15:44.440094+0000","last_peered":"2024-04-15T15:15:44.440094+0000","last_clean":"2024-04-15T15:15:44.440094+0000","last_became_active":"2024-04-15T15:15:38.571821+0000","last_became_peered":"2024-04-15T15:15:38.571821+0000","last_unstale":"2024-04-15T15:15:44.440094+0000","last_undegraded":"2024-04-15T15:15:44.440094+0000","last_fullsized":"2024-04-15T15:15:44.440094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440332+0000","last_change":"2024-04-15T15:15:37.562506+0000","last_active":"2024-04-15T15:15:44.440332+0000","last_peered":"2024-04-15T15:15:44.440332+0000","last_clean":"2024-04-15T15:15:44.440332+0000","last_became_active":"2024-04-15T15:15:37.562212+0000","last_became_peered":"2024-04-15T15:15:37.562212+0000","last_unstale":"2024-04-15T15:15:44.440332+0000","last_undegraded":"2024-04-15T15:15:44.440332+0000","last_fullsized":"2024-04-15T15:15:44.440332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440199+0000","last_change":"2024-04-15T15:15:42.656037+0000","last_active":"2024-04-15T15:15:44.440199+0000","last_peered":"2024-04-15T15:15:44.440199+0000","last_clean":"2024-04-15T15:15:44.440199+0000","last_became_active":"2024-04-15T15:15:42.655647+0000","last_became_peered":"2024-04-15T15:15:42.655647+0000","last_unstale":"2024-04-15T15:15:44.440199+0000","last_undegraded":"2024-04-15T15:15:44.440199+0000","last_fullsized":"2024-04-15T15:15:44.440199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:53:58.204060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439284+0000","last_change":"2024-04-15T15:15:38.573064+0000","last_active":"2024-04-15T15:15:44.439284+0000","last_peered":"2024-04-15T15:15:44.439284+0000","last_clean":"2024-04-15T15:15:44.439284+0000","last_became_active":"2024-04-15T15:15:38.572766+0000","last_became_peered":"2024-04-15T15:15:38.572766+0000","last_unstale":"2024-04-15T15:15:44.439284+0000","last_undegraded":"2024-04-15T15:15:44.439284+0000","last_fullsized":"2024-04-15T15:15:44.439284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:53:30.455027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439611+0000","last_change":"2024-04-15T15:15:37.565383+0000","last_active":"2024-04-15T15:15:44.439611+0000","last_peered":"2024-04-15T15:15:44.439611+0000","last_clean":"2024-04-15T15:15:44.439611+0000","last_became_active":"2024-04-15T15:15:37.561007+0000","last_became_peered":"2024-04-15T15:15:37.561007+0000","last_unstale":"2024-04-15T15:15:44.439611+0000","last_undegraded":"2024-04-15T15:15:44.439611+0000","last_fullsized":"2024-04-15T15:15:44.439611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440266+0000","last_change":"2024-04-15T15:15:42.657177+0000","last_active":"2024-04-15T15:15:44.440266+0000","last_peered":"2024-04-15T15:15:44.440266+0000","last_clean":"2024-04-15T15:15:44.440266+0000","last_became_active":"2024-04-15T15:15:42.656776+0000","last_became_peered":"2024-04-15T15:15:42.656776+0000","last_unstale":"2024-04-15T15:15:44.440266+0000","last_undegraded":"2024-04-15T15:15:44.440266+0000","last_fullsized":"2024-04-15T15:15:44.440266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439546+0000","last_change":"2024-04-15T15:15:38.569532+0000","last_active":"2024-04-15T15:15:44.439546+0000","last_peered":"2024-04-15T15:15:44.439546+0000","last_clean":"2024-04-15T15:15:44.439546+0000","last_became_active":"2024-04-15T15:15:38.569097+0000","last_became_peered":"2024-04-15T15:15:38.569097+0000","last_unstale":"2024-04-15T15:15:44.439546+0000","last_undegraded":"2024-04-15T15:15:44.439546+0000","last_fullsized":"2024-04-15T15:15:44.439546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:48.573127+0000","last_change":"2024-04-15T15:15:37.559420+0000","last_active":"2024-04-15T15:15:48.573127+0000","last_peered":"2024-04-15T15:15:48.573127+0000","last_clean":"2024-04-15T15:15:48.573127+0000","last_became_active":"2024-04-15T15:15:37.559152+0000","last_became_peered":"2024-04-15T15:15:37.559152+0000","last_unstale":"2024-04-15T15:15:48.573127+0000","last_undegraded":"2024-04-15T15:15:48.573127+0000","last_fullsized":"2024-04-15T15:15:48.573127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587812+0000","last_change":"2024-04-15T15:15:42.591766+0000","last_active":"2024-04-15T15:15:43.587812+0000","last_peered":"2024-04-15T15:15:43.587812+0000","last_clean":"2024-04-15T15:15:43.587812+0000","last_became_active":"2024-04-15T15:15:42.591354+0000","last_became_peered":"2024-04-15T15:15:42.591354+0000","last_unstale":"2024-04-15T15:15:43.587812+0000","last_undegraded":"2024-04-15T15:15:43.587812+0000","last_fullsized":"2024-04-15T15:15:43.587812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588450+0000","last_change":"2024-04-15T15:15:38.562820+0000","last_active":"2024-04-15T15:15:43.588450+0000","last_peered":"2024-04-15T15:15:43.588450+0000","last_clean":"2024-04-15T15:15:43.588450+0000","last_became_active":"2024-04-15T15:15:38.562536+0000","last_became_peered":"2024-04-15T15:15:38.562536+0000","last_unstale":"2024-04-15T15:15:43.588450+0000","last_undegraded":"2024-04-15T15:15:43.588450+0000","last_fullsized":"2024-04-15T15:15:43.588450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587577+0000","last_change":"2024-04-15T15:15:37.558919+0000","last_active":"2024-04-15T15:15:43.587577+0000","last_peered":"2024-04-15T15:15:43.587577+0000","last_clean":"2024-04-15T15:15:43.587577+0000","last_became_active":"2024-04-15T15:15:37.558592+0000","last_became_peered":"2024-04-15T15:15:37.558592+0000","last_unstale":"2024-04-15T15:15:43.587577+0000","last_undegraded":"2024-04-15T15:15:43.587577+0000","last_fullsized":"2024-04-15T15:15:43.587577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587908+0000","last_change":"2024-04-15T15:15:42.591079+0000","last_active":"2024-04-15T15:15:43.587908+0000","last_peered":"2024-04-15T15:15:43.587908+0000","last_clean":"2024-04-15T15:15:43.587908+0000","last_became_active":"2024-04-15T15:15:42.590633+0000","last_became_peered":"2024-04-15T15:15:42.590633+0000","last_unstale":"2024-04-15T15:15:43.587908+0000","last_undegraded":"2024-04-15T15:15:43.587908+0000","last_fullsized":"2024-04-15T15:15:43.587908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:33.636220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439372+0000","last_change":"2024-04-15T15:15:38.574069+0000","last_active":"2024-04-15T15:15:44.439372+0000","last_peered":"2024-04-15T15:15:44.439372+0000","last_clean":"2024-04-15T15:15:44.439372+0000","last_became_active":"2024-04-15T15:15:38.573139+0000","last_became_peered":"2024-04-15T15:15:38.573139+0000","last_unstale":"2024-04-15T15:15:44.439372+0000","last_undegraded":"2024-04-15T15:15:44.439372+0000","last_fullsized":"2024-04-15T15:15:44.439372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440496+0000","last_change":"2024-04-15T15:15:37.563424+0000","last_active":"2024-04-15T15:15:44.440496+0000","last_peered":"2024-04-15T15:15:44.440496+0000","last_clean":"2024-04-15T15:15:44.440496+0000","last_became_active":"2024-04-15T15:15:37.562924+0000","last_became_peered":"2024-04-15T15:15:37.562924+0000","last_unstale":"2024-04-15T15:15:44.440496+0000","last_undegraded":"2024-04-15T15:15:44.440496+0000","last_fullsized":"2024-04-15T15:15:44.440496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439129+0000","last_change":"2024-04-15T15:15:42.587632+0000","last_active":"2024-04-15T15:15:44.439129+0000","last_peered":"2024-04-15T15:15:44.439129+0000","last_clean":"2024-04-15T15:15:44.439129+0000","last_became_active":"2024-04-15T15:15:42.587139+0000","last_became_peered":"2024-04-15T15:15:42.587139+0000","last_unstale":"2024-04-15T15:15:44.439129+0000","last_undegraded":"2024-04-15T15:15:44.439129+0000","last_fullsized":"2024-04-15T15:15:44.439129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440397+0000","last_change":"2024-04-15T15:15:38.567493+0000","last_active":"2024-04-15T15:15:44.440397+0000","last_peered":"2024-04-15T15:15:44.440397+0000","last_clean":"2024-04-15T15:15:44.440397+0000","last_became_active":"2024-04-15T15:15:38.567188+0000","last_became_peered":"2024-04-15T15:15:38.567188+0000","last_unstale":"2024-04-15T15:15:44.440397+0000","last_undegraded":"2024-04-15T15:15:44.440397+0000","last_fullsized":"2024-04-15T15:15:44.440397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588898+0000","last_change":"2024-04-15T15:15:37.563986+0000","last_active":"2024-04-15T15:15:43.588898+0000","last_peered":"2024-04-15T15:15:43.588898+0000","last_clean":"2024-04-15T15:15:43.588898+0000","last_became_active":"2024-04-15T15:15:37.563666+0000","last_became_peered":"2024-04-15T15:15:37.563666+0000","last_unstale":"2024-04-15T15:15:43.588898+0000","last_undegraded":"2024-04-15T15:15:43.588898+0000","last_fullsized":"2024-04-15T15:15:43.588898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440053+0000","last_change":"2024-04-15T15:15:42.661474+0000","last_active":"2024-04-15T15:15:44.440053+0000","last_peered":"2024-04-15T15:15:44.440053+0000","last_clean":"2024-04-15T15:15:44.440053+0000","last_became_active":"2024-04-15T15:15:42.660740+0000","last_became_peered":"2024-04-15T15:15:42.660740+0000","last_unstale":"2024-04-15T15:15:44.440053+0000","last_undegraded":"2024-04-15T15:15:44.440053+0000","last_fullsized":"2024-04-15T15:15:44.440053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:56:48.355956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439422+0000","last_change":"2024-04-15T15:15:42.658445+0000","last_active":"2024-04-15T15:15:44.439422+0000","last_peered":"2024-04-15T15:15:44.439422+0000","last_clean":"2024-04-15T15:15:44.439422+0000","last_became_active":"2024-04-15T15:15:42.658104+0000","last_became_peered":"2024-04-15T15:15:42.658104+0000","last_unstale":"2024-04-15T15:15:44.439422+0000","last_undegraded":"2024-04-15T15:15:44.439422+0000","last_fullsized":"2024-04-15T15:15:44.439422+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586589+0000","last_change":"2024-04-15T15:15:38.562893+0000","last_active":"2024-04-15T15:15:43.586589+0000","last_peered":"2024-04-15T15:15:43.586589+0000","last_clean":"2024-04-15T15:15:43.586589+0000","last_became_active":"2024-04-15T15:15:38.562604+0000","last_became_peered":"2024-04-15T15:15:38.562604+0000","last_unstale":"2024-04-15T15:15:43.586589+0000","last_undegraded":"2024-04-15T15:15:43.586589+0000","last_fullsized":"2024-04-15T15:15:43.586589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:37.556524+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:37.556249+0000","last_became_peered":"2024-04-15T15:15:37.556249+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588777+0000","last_change":"2024-04-15T15:15:42.590110+0000","last_active":"2024-04-15T15:15:43.588777+0000","last_peered":"2024-04-15T15:15:43.588777+0000","last_clean":"2024-04-15T15:15:43.588777+0000","last_became_active":"2024-04-15T15:15:42.589355+0000","last_became_peered":"2024-04-15T15:15:42.589355+0000","last_unstale":"2024-04-15T15:15:43.588777+0000","last_undegraded":"2024-04-15T15:15:43.588777+0000","last_fullsized":"2024-04-15T15:15:43.588777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439180+0000","last_change":"2024-04-15T15:15:38.567772+0000","last_active":"2024-04-15T15:15:44.439180+0000","last_peered":"2024-04-15T15:15:44.439180+0000","last_clean":"2024-04-15T15:15:44.439180+0000","last_became_active":"2024-04-15T15:15:38.566987+0000","last_became_peered":"2024-04-15T15:15:38.566987+0000","last_unstale":"2024-04-15T15:15:44.439180+0000","last_undegraded":"2024-04-15T15:15:44.439180+0000","last_fullsized":"2024-04-15T15:15:44.439180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440394+0000","last_change":"2024-04-15T15:15:37.561996+0000","last_active":"2024-04-15T15:15:44.440394+0000","last_peered":"2024-04-15T15:15:44.440394+0000","last_clean":"2024-04-15T15:15:44.440394+0000","last_became_active":"2024-04-15T15:15:37.561696+0000","last_became_peered":"2024-04-15T15:15:37.561696+0000","last_unstale":"2024-04-15T15:15:44.440394+0000","last_undegraded":"2024-04-15T15:15:44.440394+0000","last_fullsized":"2024-04-15T15:15:44.440394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588032+0000","last_change":"2024-04-15T15:15:37.561778+0000","last_active":"2024-04-15T15:15:43.588032+0000","last_peered":"2024-04-15T15:15:43.588032+0000","last_clean":"2024-04-15T15:15:43.588032+0000","last_became_active":"2024-04-15T15:15:37.561489+0000","last_became_peered":"2024-04-15T15:15:37.561489+0000","last_unstale":"2024-04-15T15:15:43.588032+0000","last_undegraded":"2024-04-15T15:15:43.588032+0000","last_fullsized":"2024-04-15T15:15:43.588032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439557+0000","last_change":"2024-04-15T15:15:38.570137+0000","last_active":"2024-04-15T15:15:44.439557+0000","last_peered":"2024-04-15T15:15:44.439557+0000","last_clean":"2024-04-15T15:15:44.439557+0000","last_became_active":"2024-04-15T15:15:38.569657+0000","last_became_peered":"2024-04-15T15:15:38.569657+0000","last_unstale":"2024-04-15T15:15:44.439557+0000","last_undegraded":"2024-04-15T15:15:44.439557+0000","last_fullsized":"2024-04-15T15:15:44.439557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769366+0000","last_change":"2024-04-15T15:15:42.584680+0000","last_active":"2024-04-15T15:15:43.769366+0000","last_peered":"2024-04-15T15:15:43.769366+0000","last_clean":"2024-04-15T15:15:43.769366+0000","last_became_active":"2024-04-15T15:15:42.584312+0000","last_became_peered":"2024-04-15T15:15:42.584312+0000","last_unstale":"2024-04-15T15:15:43.769366+0000","last_undegraded":"2024-04-15T15:15:43.769366+0000","last_fullsized":"2024-04-15T15:15:43.769366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:39.473475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440046+0000","last_change":"2024-04-15T15:15:37.565397+0000","last_active":"2024-04-15T15:15:44.440046+0000","last_peered":"2024-04-15T15:15:44.440046+0000","last_clean":"2024-04-15T15:15:44.440046+0000","last_became_active":"2024-04-15T15:15:37.565018+0000","last_became_peered":"2024-04-15T15:15:37.565018+0000","last_unstale":"2024-04-15T15:15:44.440046+0000","last_undegraded":"2024-04-15T15:15:44.440046+0000","last_fullsized":"2024-04-15T15:15:44.440046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587469+0000","last_change":"2024-04-15T15:15:38.558591+0000","last_active":"2024-04-15T15:15:43.587469+0000","last_peered":"2024-04-15T15:15:43.587469+0000","last_clean":"2024-04-15T15:15:43.587469+0000","last_became_active":"2024-04-15T15:15:38.558044+0000","last_became_peered":"2024-04-15T15:15:38.558044+0000","last_unstale":"2024-04-15T15:15:43.587469+0000","last_undegraded":"2024-04-15T15:15:43.587469+0000","last_fullsized":"2024-04-15T15:15:43.587469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:56.009007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439916+0000","last_change":"2024-04-15T15:15:42.587509+0000","last_active":"2024-04-15T15:15:44.439916+0000","last_peered":"2024-04-15T15:15:44.439916+0000","last_clean":"2024-04-15T15:15:44.439916+0000","last_became_active":"2024-04-15T15:15:42.587097+0000","last_became_peered":"2024-04-15T15:15:42.587097+0000","last_unstale":"2024-04-15T15:15:44.439916+0000","last_undegraded":"2024-04-15T15:15:44.439916+0000","last_fullsized":"2024-04-15T15:15:44.439916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:55:01.099477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588741+0000","last_change":"2024-04-15T15:15:38.566026+0000","last_active":"2024-04-15T15:15:43.588741+0000","last_peered":"2024-04-15T15:15:43.588741+0000","last_clean":"2024-04-15T15:15:43.588741+0000","last_became_active":"2024-04-15T15:15:38.565726+0000","last_became_peered":"2024-04-15T15:15:38.565726+0000","last_unstale":"2024-04-15T15:15:43.588741+0000","last_undegraded":"2024-04-15T15:15:43.588741+0000","last_fullsized":"2024-04-15T15:15:43.588741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769196+0000","last_change":"2024-04-15T15:15:38.567487+0000","last_active":"2024-04-15T15:15:43.769196+0000","last_peered":"2024-04-15T15:15:43.769196+0000","last_clean":"2024-04-15T15:15:43.769196+0000","last_became_active":"2024-04-15T15:15:38.567201+0000","last_became_peered":"2024-04-15T15:15:38.567201+0000","last_unstale":"2024-04-15T15:15:43.769196+0000","last_undegraded":"2024-04-15T15:15:43.769196+0000","last_fullsized":"2024-04-15T15:15:43.769196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440239+0000","last_change":"2024-04-15T15:15:38.569794+0000","last_active":"2024-04-15T15:15:44.440239+0000","last_peered":"2024-04-15T15:15:44.440239+0000","last_clean":"2024-04-15T15:15:44.440239+0000","last_became_active":"2024-04-15T15:15:38.569510+0000","last_became_peered":"2024-04-15T15:15:38.569510+0000","last_unstale":"2024-04-15T15:15:44.440239+0000","last_undegraded":"2024-04-15T15:15:44.440239+0000","last_fullsized":"2024-04-15T15:15:44.440239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439457+0000","last_change":"2024-04-15T15:15:38.563215+0000","last_active":"2024-04-15T15:15:44.439457+0000","last_peered":"2024-04-15T15:15:44.439457+0000","last_clean":"2024-04-15T15:15:44.439457+0000","last_became_active":"2024-04-15T15:15:38.562942+0000","last_became_peered":"2024-04-15T15:15:38.562942+0000","last_unstale":"2024-04-15T15:15:44.439457+0000","last_undegraded":"2024-04-15T15:15:44.439457+0000","last_fullsized":"2024-04-15T15:15:44.439457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440355+0000","last_change":"2024-04-15T15:15:38.567774+0000","last_active":"2024-04-15T15:15:44.440355+0000","last_peered":"2024-04-15T15:15:44.440355+0000","last_clean":"2024-04-15T15:15:44.440355+0000","last_became_active":"2024-04-15T15:15:38.567471+0000","last_became_peered":"2024-04-15T15:15:38.567471+0000","last_unstale":"2024-04-15T15:15:44.440355+0000","last_undegraded":"2024-04-15T15:15:44.440355+0000","last_fullsized":"2024-04-15T15:15:44.440355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587284+0000","last_change":"2024-04-15T15:15:38.562963+0000","last_active":"2024-04-15T15:15:43.587284+0000","last_peered":"2024-04-15T15:15:43.587284+0000","last_clean":"2024-04-15T15:15:43.587284+0000","last_became_active":"2024-04-15T15:15:38.562611+0000","last_became_peered":"2024-04-15T15:15:38.562611+0000","last_unstale":"2024-04-15T15:15:43.587284+0000","last_undegraded":"2024-04-15T15:15:43.587284+0000","last_fullsized":"2024-04-15T15:15:43.587284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588859+0000","last_change":"2024-04-15T15:15:38.566158+0000","last_active":"2024-04-15T15:15:43.588859+0000","last_peered":"2024-04-15T15:15:43.588859+0000","last_clean":"2024-04-15T15:15:43.588859+0000","last_became_active":"2024-04-15T15:15:38.565859+0000","last_became_peered":"2024-04-15T15:15:38.565859+0000","last_unstale":"2024-04-15T15:15:43.588859+0000","last_undegraded":"2024-04-15T15:15:43.588859+0000","last_fullsized":"2024-04-15T15:15:43.588859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440318+0000","last_change":"2024-04-15T15:15:38.566715+0000","last_active":"2024-04-15T15:15:44.440318+0000","last_peered":"2024-04-15T15:15:44.440318+0000","last_clean":"2024-04-15T15:15:44.440318+0000","last_became_active":"2024-04-15T15:15:38.565677+0000","last_became_peered":"2024-04-15T15:15:38.565677+0000","last_unstale":"2024-04-15T15:15:44.440318+0000","last_undegraded":"2024-04-15T15:15:44.440318+0000","last_fullsized":"2024-04-15T15:15:44.440318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439302+0000","last_change":"2024-04-15T15:15:38.565562+0000","last_active":"2024-04-15T15:15:44.439302+0000","last_peered":"2024-04-15T15:15:44.439302+0000","last_clean":"2024-04-15T15:15:44.439302+0000","last_became_active":"2024-04-15T15:15:38.564779+0000","last_became_peered":"2024-04-15T15:15:38.564779+0000","last_unstale":"2024-04-15T15:15:44.439302+0000","last_undegraded":"2024-04-15T15:15:44.439302+0000","last_fullsized":"2024-04-15T15:15:44.439302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440302+0000","last_change":"2024-04-15T15:15:38.557876+0000","last_active":"2024-04-15T15:15:44.440302+0000","last_peered":"2024-04-15T15:15:44.440302+0000","last_clean":"2024-04-15T15:15:44.440302+0000","last_became_active":"2024-04-15T15:15:38.557568+0000","last_became_peered":"2024-04-15T15:15:38.557568+0000","last_unstale":"2024-04-15T15:15:44.440302+0000","last_undegraded":"2024-04-15T15:15:44.440302+0000","last_fullsized":"2024-04-15T15:15:44.440302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440139+0000","last_change":"2024-04-15T15:15:38.568437+0000","last_active":"2024-04-15T15:15:44.440139+0000","last_peered":"2024-04-15T15:15:44.440139+0000","last_clean":"2024-04-15T15:15:44.440139+0000","last_became_active":"2024-04-15T15:15:38.567071+0000","last_became_peered":"2024-04-15T15:15:38.567071+0000","last_unstale":"2024-04-15T15:15:44.440139+0000","last_undegraded":"2024-04-15T15:15:44.440139+0000","last_fullsized":"2024-04-15T15:15:44.440139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439337+0000","last_change":"2024-04-15T15:15:38.572158+0000","last_active":"2024-04-15T15:15:44.439337+0000","last_peered":"2024-04-15T15:15:44.439337+0000","last_clean":"2024-04-15T15:15:44.439337+0000","last_became_active":"2024-04-15T15:15:38.571868+0000","last_became_peered":"2024-04-15T15:15:38.571868+0000","last_unstale":"2024-04-15T15:15:44.439337+0000","last_undegraded":"2024-04-15T15:15:44.439337+0000","last_fullsized":"2024-04-15T15:15:44.439337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588528+0000","last_change":"2024-04-15T15:15:38.556771+0000","last_active":"2024-04-15T15:15:43.588528+0000","last_peered":"2024-04-15T15:15:43.588528+0000","last_clean":"2024-04-15T15:15:43.588528+0000","last_became_active":"2024-04-15T15:15:38.556456+0000","last_became_peered":"2024-04-15T15:15:38.556456+0000","last_unstale":"2024-04-15T15:15:43.588528+0000","last_undegraded":"2024-04-15T15:15:43.588528+0000","last_fullsized":"2024-04-15T15:15:43.588528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439982+0000","last_change":"2024-04-15T15:15:38.565285+0000","last_active":"2024-04-15T15:15:44.439982+0000","last_peered":"2024-04-15T15:15:44.439982+0000","last_clean":"2024-04-15T15:15:44.439982+0000","last_became_active":"2024-04-15T15:15:38.564505+0000","last_became_peered":"2024-04-15T15:15:38.564505+0000","last_unstale":"2024-04-15T15:15:44.439982+0000","last_undegraded":"2024-04-15T15:15:44.439982+0000","last_fullsized":"2024-04-15T15:15:44.439982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:44.185650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769488+0000","last_change":"2024-04-15T15:15:38.562279+0000","last_active":"2024-04-15T15:15:43.769488+0000","last_peered":"2024-04-15T15:15:43.769488+0000","last_clean":"2024-04-15T15:15:43.769488+0000","last_became_active":"2024-04-15T15:15:38.561971+0000","last_became_peered":"2024-04-15T15:15:38.561971+0000","last_unstale":"2024-04-15T15:15:43.769488+0000","last_undegraded":"2024-04-15T15:15:43.769488+0000","last_fullsized":"2024-04-15T15:15:43.769488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440908+0000","last_change":"2024-04-15T15:15:38.569181+0000","last_active":"2024-04-15T15:15:44.440908+0000","last_peered":"2024-04-15T15:15:44.440908+0000","last_clean":"2024-04-15T15:15:44.440908+0000","last_became_active":"2024-04-15T15:15:38.566602+0000","last_became_peered":"2024-04-15T15:15:38.566602+0000","last_unstale":"2024-04-15T15:15:44.440908+0000","last_undegraded":"2024-04-15T15:15:44.440908+0000","last_fullsized":"2024-04-15T15:15:44.440908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439191+0000","last_change":"2024-04-15T15:15:38.568218+0000","last_active":"2024-04-15T15:15:44.439191+0000","last_peered":"2024-04-15T15:15:44.439191+0000","last_clean":"2024-04-15T15:15:44.439191+0000","last_became_active":"2024-04-15T15:15:38.566966+0000","last_became_peered":"2024-04-15T15:15:38.566966+0000","last_unstale":"2024-04-15T15:15:44.439191+0000","last_undegraded":"2024-04-15T15:15:44.439191+0000","last_fullsized":"2024-04-15T15:15:44.439191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586673+0000","last_change":"2024-04-15T15:15:38.563656+0000","last_active":"2024-04-15T15:15:43.586673+0000","last_peered":"2024-04-15T15:15:43.586673+0000","last_clean":"2024-04-15T15:15:43.586673+0000","last_became_active":"2024-04-15T15:15:38.563380+0000","last_became_peered":"2024-04-15T15:15:38.563380+0000","last_unstale":"2024-04-15T15:15:43.586673+0000","last_undegraded":"2024-04-15T15:15:43.586673+0000","last_fullsized":"2024-04-15T15:15:43.586673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439190+0000","last_change":"2024-04-15T15:15:38.558804+0000","last_active":"2024-04-15T15:15:44.439190+0000","last_peered":"2024-04-15T15:15:44.439190+0000","last_clean":"2024-04-15T15:15:44.439190+0000","last_became_active":"2024-04-15T15:15:38.558374+0000","last_became_peered":"2024-04-15T15:15:38.558374+0000","last_unstale":"2024-04-15T15:15:44.439190+0000","last_undegraded":"2024-04-15T15:15:44.439190+0000","last_fullsized":"2024-04-15T15:15:44.439190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:35.448266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439861+0000","last_change":"2024-04-15T15:15:38.564455+0000","last_active":"2024-04-15T15:15:44.439861+0000","last_peered":"2024-04-15T15:15:44.439861+0000","last_clean":"2024-04-15T15:15:44.439861+0000","last_became_active":"2024-04-15T15:15:38.563328+0000","last_became_peered":"2024-04-15T15:15:38.563328+0000","last_unstale":"2024-04-15T15:15:44.439861+0000","last_undegraded":"2024-04-15T15:15:44.439861+0000","last_fullsized":"2024-04-15T15:15:44.439861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588424+0000","last_change":"2024-04-15T15:15:38.563202+0000","last_active":"2024-04-15T15:15:43.588424+0000","last_peered":"2024-04-15T15:15:43.588424+0000","last_clean":"2024-04-15T15:15:43.588424+0000","last_became_active":"2024-04-15T15:15:38.562862+0000","last_became_peered":"2024-04-15T15:15:38.562862+0000","last_unstale":"2024-04-15T15:15:43.588424+0000","last_undegraded":"2024-04-15T15:15:43.588424+0000","last_fullsized":"2024-04-15T15:15:43.588424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588478+0000","last_change":"2024-04-15T15:15:38.570615+0000","last_active":"2024-04-15T15:15:43.588478+0000","last_peered":"2024-04-15T15:15:43.588478+0000","last_clean":"2024-04-15T15:15:43.588478+0000","last_became_active":"2024-04-15T15:15:38.570340+0000","last_became_peered":"2024-04-15T15:15:38.570340+0000","last_unstale":"2024-04-15T15:15:43.588478+0000","last_undegraded":"2024-04-15T15:15:43.588478+0000","last_fullsized":"2024-04-15T15:15:43.588478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588803+0000","last_change":"2024-04-15T15:15:38.567756+0000","last_active":"2024-04-15T15:15:43.588803+0000","last_peered":"2024-04-15T15:15:43.588803+0000","last_clean":"2024-04-15T15:15:43.588803+0000","last_became_active":"2024-04-15T15:15:38.567455+0000","last_became_peered":"2024-04-15T15:15:38.567455+0000","last_unstale":"2024-04-15T15:15:43.588803+0000","last_undegraded":"2024-04-15T15:15:43.588803+0000","last_fullsized":"2024-04-15T15:15:43.588803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440004+0000","last_change":"2024-04-15T15:15:38.574989+0000","last_active":"2024-04-15T15:15:44.440004+0000","last_peered":"2024-04-15T15:15:44.440004+0000","last_clean":"2024-04-15T15:15:44.440004+0000","last_became_active":"2024-04-15T15:15:38.573776+0000","last_became_peered":"2024-04-15T15:15:38.573776+0000","last_unstale":"2024-04-15T15:15:44.440004+0000","last_undegraded":"2024-04-15T15:15:44.440004+0000","last_fullsized":"2024-04-15T15:15:44.440004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:41.104693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438915+0000","last_change":"2024-04-15T15:15:38.573020+0000","last_active":"2024-04-15T15:15:44.438915+0000","last_peered":"2024-04-15T15:15:44.438915+0000","last_clean":"2024-04-15T15:15:44.438915+0000","last_became_active":"2024-04-15T15:15:38.572715+0000","last_became_peered":"2024-04-15T15:15:38.572715+0000","last_unstale":"2024-04-15T15:15:44.438915+0000","last_undegraded":"2024-04-15T15:15:44.438915+0000","last_fullsized":"2024-04-15T15:15:44.438915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439587+0000","last_change":"2024-04-15T15:15:38.568749+0000","last_active":"2024-04-15T15:15:44.439587+0000","last_peered":"2024-04-15T15:15:44.439587+0000","last_clean":"2024-04-15T15:15:44.439587+0000","last_became_active":"2024-04-15T15:15:38.568488+0000","last_became_peered":"2024-04-15T15:15:38.568488+0000","last_unstale":"2024-04-15T15:15:44.439587+0000","last_undegraded":"2024-04-15T15:15:44.439587+0000","last_fullsized":"2024-04-15T15:15:44.439587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440025+0000","last_change":"2024-04-15T15:15:38.572663+0000","last_active":"2024-04-15T15:15:44.440025+0000","last_peered":"2024-04-15T15:15:44.440025+0000","last_clean":"2024-04-15T15:15:44.440025+0000","last_became_active":"2024-04-15T15:15:38.572349+0000","last_became_peered":"2024-04-15T15:15:38.572349+0000","last_unstale":"2024-04-15T15:15:44.440025+0000","last_undegraded":"2024-04-15T15:15:44.440025+0000","last_fullsized":"2024-04-15T15:15:44.440025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439869+0000","last_change":"2024-04-15T15:15:38.567565+0000","last_active":"2024-04-15T15:15:44.439869+0000","last_peered":"2024-04-15T15:15:44.439869+0000","last_clean":"2024-04-15T15:15:44.439869+0000","last_became_active":"2024-04-15T15:15:38.567300+0000","last_became_peered":"2024-04-15T15:15:38.567300+0000","last_unstale":"2024-04-15T15:15:44.439869+0000","last_undegraded":"2024-04-15T15:15:44.439869+0000","last_fullsized":"2024-04-15T15:15:44.439869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440752+0000","last_change":"2024-04-15T15:15:38.578049+0000","last_active":"2024-04-15T15:15:44.440752+0000","last_peered":"2024-04-15T15:15:44.440752+0000","last_clean":"2024-04-15T15:15:44.440752+0000","last_became_active":"2024-04-15T15:15:38.577801+0000","last_became_peered":"2024-04-15T15:15:38.577801+0000","last_unstale":"2024-04-15T15:15:44.440752+0000","last_undegraded":"2024-04-15T15:15:44.440752+0000","last_fullsized":"2024-04-15T15:15:44.440752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439206+0000","last_change":"2024-04-15T15:15:38.561841+0000","last_active":"2024-04-15T15:15:44.439206+0000","last_peered":"2024-04-15T15:15:44.439206+0000","last_clean":"2024-04-15T15:15:44.439206+0000","last_became_active":"2024-04-15T15:15:38.561554+0000","last_became_peered":"2024-04-15T15:15:38.561554+0000","last_unstale":"2024-04-15T15:15:44.439206+0000","last_undegraded":"2024-04-15T15:15:44.439206+0000","last_fullsized":"2024-04-15T15:15:44.439206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587614+0000","last_change":"2024-04-15T15:15:38.560883+0000","last_active":"2024-04-15T15:15:43.587614+0000","last_peered":"2024-04-15T15:15:43.587614+0000","last_clean":"2024-04-15T15:15:43.587614+0000","last_became_active":"2024-04-15T15:15:38.560562+0000","last_became_peered":"2024-04-15T15:15:38.560562+0000","last_unstale":"2024-04-15T15:15:43.587614+0000","last_undegraded":"2024-04-15T15:15:43.587614+0000","last_fullsized":"2024-04-15T15:15:43.587614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439744+0000","last_change":"2024-04-15T15:15:38.565038+0000","last_active":"2024-04-15T15:15:44.439744+0000","last_peered":"2024-04-15T15:15:44.439744+0000","last_clean":"2024-04-15T15:15:44.439744+0000","last_became_active":"2024-04-15T15:15:38.564551+0000","last_became_peered":"2024-04-15T15:15:38.564551+0000","last_unstale":"2024-04-15T15:15:44.439744+0000","last_undegraded":"2024-04-15T15:15:44.439744+0000","last_fullsized":"2024-04-15T15:15:44.439744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542154,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":104213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":695037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":104087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":104247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":868352,"data_stored":695093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:03.017 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:16:03.018 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T15:16:03.018 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-15T15:16:03.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-15T15:16:03.018 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-15T15:16:03.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.155+0000 7fae8c235640 1 Processor -- start 2024-04-15T15:16:03.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.155+0000 7fae8c235640 1 -- start start 2024-04-15T15:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae8c235640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 0x7fae84160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae8c235640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae84070720 con 0x7fae8415fb30 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae8c235640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae84070a90 con 0x7fae8409a9d0 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae8c235640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae8409ba90 con 0x7fae840701f0 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae8a7d2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fae8415fb30 0x7fae840d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55574/0 (socket says 172.21.15.46:55574) 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1127214812 learned_addr learned my addr 172.21.15.46:0/1127214812 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae897d0640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 0x7fae84160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994247277 0 0) 0x7fae84070720 con 0x7fae8415fb30 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.156+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae70003520 con 0x7fae8415fb30 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3154993576 0 0) 0x7fae70003520 con 0x7fae8415fb30 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 msgr2=0x7fae84160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 0x7fae84160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 >> v1:172.21.15.78:6789/0 conn(0x7fae8409a9d0 legacy=0x7fae8409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae84070d70 con 0x7fae8415fb30 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fae7c0019f0 con 0x7fae8415fb30 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 >> v1:172.21.15.46:6789/0 conn(0x7fae8415fb30 legacy=0x7fae840d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 0x7fae700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70003f60 0x7fae70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae70004d40 0x7fae700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae7000ba70 con 0x7fae70004d40 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fae7000bc20 con 0x7fae70003f60 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae7000be20 con 0x7fae840701f0 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae70004d40 0x7fae700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae89fd1640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 0x7fae700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae897d0640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70003f60 0x7fae70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 msgr2=0x7fae700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840701f0 0x7fae700039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70003f60 msgr2=0x7fae70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70003f60 0x7fae70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.157+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1127214812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae70003520 con 0x7fae70004d40 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.158+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae70004d40 0x7fae700051c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00a850 tx=0x7fae70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76d42812ea0eb66 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.158+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00a4a0 con 0x7fae70004d40 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.158+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae7c034650 con 0x7fae70004d40 2024-04-15T15:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.158+0000 7fae88fcf640 1 -- 172.21.15.46:0/1127214812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c034a80 con 0x7fae70004d40 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8c235640 1 -- 172.21.15.46:0/1127214812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae70004d40 msgr2=0x7fae700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8c235640 1 --2- 172.21.15.46:0/1127214812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae70004d40 0x7fae700051c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00a850 tx=0x7fae70005830 comp rx=0 tx=0).stop 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1127214812 reap_dead start 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8c235640 1 -- 172.21.15.46:0/1127214812 shutdown_connections 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8c235640 1 -- 172.21.15.46:0/1127214812 >> 172.21.15.46:0/1127214812 conn(0x7fae840d2460 msgr2=0x7fae840d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8c235640 1 -- 172.21.15.46:0/1127214812 shutdown_connections 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.159+0000 7fae8c235640 1 -- 172.21.15.46:0/1127214812 wait complete. 2024-04-15T15:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.160+0000 7fae8c235640 1 Processor -- start 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.160+0000 7fae8c235640 1 -- start start 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8c235640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70004d40 0x7fae8415bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8c235640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840af250 0x7fae84150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8c235640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 0x7fae84160e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8c235640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae8415ffc0 con 0x7fae841609a0 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8c235640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fae840d08c0 con 0x7fae70004d40 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8c235640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae840b1c60 con 0x7fae840af250 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 0x7fae84160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 0x7fae84160e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42032/0 (socket says 172.21.15.46:42032) 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1343893764 learned_addr learned my addr 172.21.15.46:0/1343893764 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae897d0640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840af250 0x7fae84150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae89fd1640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70004d40 0x7fae8415bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840af250 msgr2=0x7fae84150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840af250 0x7fae84150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70004d40 msgr2=0x7fae8415bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70004d40 0x7fae8415bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 -- 172.21.15.46:0/1343893764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae840c7460 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.161+0000 7fae8a7d2640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 0x7fae84160e50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae7c001cf0 tx=0x7fae7c001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.162+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c001250 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.162+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae7c044070 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.162+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae840af7c0 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.162+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c034a00 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.162+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae840afd80 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.163+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae840689e0 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.163+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae7c034ba0 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.164+0000 7fae7a7fc640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae6806ca90 0x7fae6806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.164+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fae7c0ae0a0 con 0x7fae841609a0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.164+0000 7fae89fd1640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae6806ca90 0x7fae6806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.164+0000 7fae89fd1640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae6806ca90 0x7fae6806ef50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae800017e0 tx=0x7fae80009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.168+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae7c07c190 con 0x7fae841609a0 2024-04-15T15:16:03.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.403+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fae84161d80 con 0x7fae841609a0 2024-04-15T15:16:03.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.406+0000 7fae7a7fc640 1 -- 172.21.15.46:0/1343893764 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174136 (secure 0 0 0) 0x7fae7c07ba90 con 0x7fae841609a0 2024-04-15T15:16:03.410 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae6806ca90 msgr2=0x7fae6806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae6806ca90 0x7fae6806ef50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fae800017e0 tx=0x7fae80009040 comp rx=0 tx=0).stop 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 msgr2=0x7fae84160e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 0x7fae84160e50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fae7c001cf0 tx=0x7fae7c001dc0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 shutdown_connections 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae6806ca90 0x7fae6806ef50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae841609a0 0x7fae84160e50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae840af250 0x7fae84150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 --2- 172.21.15.46:0/1343893764 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae70004d40 0x7fae8415bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 >> 172.21.15.46:0/1343893764 conn(0x7fae840d2460 msgr2=0x7fae840d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 shutdown_connections 2024-04-15T15:16:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.409+0000 7fae8c235640 1 -- 172.21.15.46:0/1343893764 wait complete. 2024-04-15T15:16:03.422 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"flags":0,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1451944854},{"type":"v1","addr":"172.21.15.78:6833","nonce":1451944854}]},"active_addr":"172.21.15.78:6833/1451944854","active_change":"2024-04-15T15:15:18.343694+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2531817580}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":361250512}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":4012681745}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3423934897}]}]} 2024-04-15T15:16:03.425 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-15T15:16:03.425 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-15T15:16:03.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.528+0000 7f029fd24640 1 Processor -- start 2024-04-15T15:16:03.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.528+0000 7f029fd24640 1 -- start start 2024-04-15T15:16:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029fd24640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980cf600 0x7f02980cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029fd24640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02980af4e0 0x7f02980af970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029fd24640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02980ae440 con 0x7f029815ed00 2024-04-15T15:16:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029fd24640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0298071030 con 0x7f02980af4e0 2024-04-15T15:16:03.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029fd24640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f02980cffa0 con 0x7f02980cf600 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029dac0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f029815ed00 0x7f0298070920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55580/0 (socket says 172.21.15.46:55580) 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029e2c1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02980af4e0 0x7f02980af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029dac0640 1 -- 172.21.15.46:0/1022062106 learned_addr learned my addr 172.21.15.46:0/1022062106 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029d2bf640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980cf600 0x7f02980cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029e2c1640 1 -- 172.21.15.46:0/1022062106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980cf600 msgr2=0x7f02980cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029e2c1640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980cf600 0x7f02980cfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029e2c1640 1 -- 172.21.15.46:0/1022062106 >> v1:172.21.15.46:6789/0 conn(0x7f029815ed00 legacy=0x7f0298070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029e2c1640 1 -- 172.21.15.46:0/1022062106 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02980ae720 con 0x7f02980af4e0 2024-04-15T15:16:03.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.529+0000 7f029d2bf640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980cf600 0x7f02980cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.530+0000 7f029e2c1640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02980af4e0 0x7f02980af970 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f02880097a0 tx=0x7f028802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f8817d8c19f940 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.530+0000 7f029cabe640 1 -- 172.21.15.46:0/1022062106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288035020 con 0x7f02980af4e0 2024-04-15T15:16:03.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.530+0000 7f029cabe640 1 -- 172.21.15.46:0/1022062106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0288031830 con 0x7f02980af4e0 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.530+0000 7f029cabe640 1 -- 172.21.15.46:0/1022062106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288031c00 con 0x7f02980af4e0 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 -- 172.21.15.46:0/1022062106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02980af4e0 msgr2=0x7f02980af970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02980af4e0 0x7f02980af970 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f02880097a0 tx=0x7f028802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 -- 172.21.15.46:0/1022062106 shutdown_connections 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02980af4e0 0x7f02980af970 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 --2- 172.21.15.46:0/1022062106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980cf600 0x7f02980cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 -- 172.21.15.46:0/1022062106 >> 172.21.15.46:0/1022062106 conn(0x7f029809a9d0 msgr2=0x7f02980d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 -- 172.21.15.46:0/1022062106 shutdown_connections 2024-04-15T15:16:03.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.531+0000 7f029fd24640 1 -- 172.21.15.46:0/1022062106 wait complete. 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.532+0000 7f029fd24640 1 Processor -- start 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.532+0000 7f029fd24640 1 -- start start 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029fd24640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980af4e0 0x7f029813b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029fd24640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02980cf600 0x7f0298138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029fd24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 0x7f0298139500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029fd24640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02980aefc0 con 0x7f029815ed00 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029fd24640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f02980aece0 con 0x7f02980cf600 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029fd24640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f02980701a0 con 0x7f02980af4e0 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 0x7f0298139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 0x7f0298139500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42048/0 (socket says 172.21.15.46:42048) 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 -- 172.21.15.46:0/28878250 learned_addr learned my addr 172.21.15.46:0/28878250 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:03.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029d2bf640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02980cf600 0x7f0298138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029dac0640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980af4e0 0x7f029813b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 -- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980af4e0 msgr2=0x7f029813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980af4e0 0x7f029813b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 -- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02980cf600 msgr2=0x7f0298138fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02980cf600 0x7f0298138fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029e2c1640 1 -- 172.21.15.46:0/28878250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02980b6ae0 con 0x7f029815ed00 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.533+0000 7f029dac0640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980af4e0 0x7f029813b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.534+0000 7f029e2c1640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 0x7f0298139500 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0288034040 tx=0x7f02880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.534+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288040070 con 0x7f029815ed00 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.534+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02880398e0 con 0x7f029815ed00 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.534+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288035040 con 0x7f029815ed00 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.534+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02980b5d20 con 0x7f029815ed00 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.534+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02980b62e0 con 0x7f029815ed00 2024-04-15T15:16:03.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.535+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02980690f0 con 0x7f029815ed00 2024-04-15T15:16:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.535+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0288038020 con 0x7f029815ed00 2024-04-15T15:16:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.536+0000 7f02967fc640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028406c8f0 0x7f028406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.536+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f02880ad6d0 con 0x7f029815ed00 2024-04-15T15:16:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.536+0000 7f029dac0640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028406c8f0 0x7f028406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.536+0000 7f029dac0640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028406c8f0 0x7f028406edb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02900012e0 tx=0x7f0290009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.540+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f028807b850 con 0x7f029815ed00 2024-04-15T15:16:03.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.729+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f029806f360 con 0x7f029815ed00 2024-04-15T15:16:03.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.730+0000 7f02967fc640 1 -- 172.21.15.46:0/28878250 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f028807b150 con 0x7f029815ed00 2024-04-15T15:16:03.732 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:03.732 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:03.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028406c8f0 msgr2=0x7f028406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028406c8f0 0x7f028406edb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02900012e0 tx=0x7f0290009040 comp rx=0 tx=0).stop 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 msgr2=0x7f0298139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 0x7f0298139500 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0288034040 tx=0x7f02880046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 shutdown_connections 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f028406c8f0 0x7f028406edb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f029815ed00 0x7f0298139500 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02980cf600 0x7f0298138fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 --2- 172.21.15.46:0/28878250 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f02980af4e0 0x7f029813b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 >> 172.21.15.46:0/28878250 conn(0x7f029809a9d0 msgr2=0x7f029806f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 shutdown_connections 2024-04-15T15:16:03.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.732+0000 7f029fd24640 1 -- 172.21.15.46:0/28878250 wait complete. 2024-04-15T15:16:03.746 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-15T15:16:03.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:03.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.850+0000 7f28c6f9d640 1 Processor -- start 2024-04-15T15:16:03.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.850+0000 7f28c6f9d640 1 -- start start 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28c6f9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28c00d3240 0x7f28c00d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28c6f9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c00b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28c6f9d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c00d2480 con 0x7f28c015e790 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28c6f9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f28c00d2db0 con 0x7f28c00d3240 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28c6f9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f28c00d2760 con 0x7f28c00d1440 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28c00d3240 0x7f28c00d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f28c00d3240 0x7f28c00d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42050/0 (socket says 172.21.15.46:42050) 2024-04-15T15:16:03.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 -- 172.21.15.46:0/97996075 learned_addr learned my addr 172.21.15.46:0/97996075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28c553a640 1 --2- 172.21.15.46:0/97996075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c00b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 -- 172.21.15.46:0/97996075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 msgr2=0x7f28c00b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 --2- 172.21.15.46:0/97996075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c00b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 -- 172.21.15.46:0/97996075 >> v1:172.21.15.78:6789/0 conn(0x7f28c015e790 legacy=0x7f28c00cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.851+0000 7f28bffff640 1 -- 172.21.15.46:0/97996075 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c00d2a40 con 0x7f28c00d3240 2024-04-15T15:16:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28bffff640 1 --2- 172.21.15.46:0/97996075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c00d3240 0x7f28c00d0f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009e50 tx=0x7f28b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219cd4ab7829a034 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28bf7fe640 1 -- 172.21.15.46:0/97996075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b002cb00 con 0x7f28c00d3240 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28bf7fe640 1 -- 172.21.15.46:0/97996075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b002cca0 con 0x7f28c00d3240 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28bf7fe640 1 -- 172.21.15.46:0/97996075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0034610 con 0x7f28c00d3240 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28c6f9d640 1 -- 172.21.15.46:0/97996075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c00d3240 msgr2=0x7f28c00d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/97996075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c00d3240 0x7f28c00d0f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28b0009e50 tx=0x7f28b002bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28c6f9d640 1 -- 172.21.15.46:0/97996075 shutdown_connections 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/97996075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c00b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/97996075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c00d3240 0x7f28c00d0f00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.852+0000 7f28c6f9d640 1 -- 172.21.15.46:0/97996075 >> 172.21.15.46:0/97996075 conn(0x7f28c008bf60 msgr2=0x7f28c00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.853+0000 7f28c6f9d640 1 -- 172.21.15.46:0/97996075 shutdown_connections 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.853+0000 7f28c6f9d640 1 -- 172.21.15.46:0/97996075 wait complete. 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.854+0000 7f28c6f9d640 1 Processor -- start 2024-04-15T15:16:03.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.854+0000 7f28c6f9d640 1 -- start start 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c0152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28c00d3240 0x7f28c00c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 0x7f28c00ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f28c00d5180 con 0x7f28c015e790 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f28c00d0430 con 0x7f28c00d3240 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f28c00d5300 con 0x7f28c00d1440 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 0x7f28c00ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 0x7f28c00ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42052/0 (socket says 172.21.15.46:42052) 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 -- 172.21.15.46:0/4205594010 learned_addr learned my addr 172.21.15.46:0/4205594010 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28bffff640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28c00d3240 0x7f28c00c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c4d39640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c0152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 -- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 msgr2=0x7f28c0152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c0152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 -- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28c00d3240 msgr2=0x7f28c00c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28c00d3240 0x7f28c00c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 -- 172.21.15.46:0/4205594010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c01f89d0 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c553a640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 0x7f28c00ca190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f28b800d800 tx=0x7f28b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b800b6c0 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b8019070 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b800bac0 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c01f7c10 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.855+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c01f81d0 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.856+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c00689e0 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.857+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f28b800bc60 con 0x7f28c015e790 2024-04-15T15:16:03.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.857+0000 7f28bd7fa640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f28a006c8f0 0x7f28a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.857+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f28b808c130 con 0x7f28c015e790 2024-04-15T15:16:03.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.858+0000 7f28c4d39640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f28a006c8f0 0x7f28a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.858+0000 7f28c4d39640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f28a006c8f0 0x7f28a006edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f28a80016e0 tx=0x7f28a8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:03.862+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f28b8059c60 con 0x7f28c015e790 2024-04-15T15:16:04.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.047+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28c00d1ce0 con 0x7f28c015e790 2024-04-15T15:16:04.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.048+0000 7f28bd7fa640 1 -- 172.21.15.46:0/4205594010 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7f28b805df60 con 0x7f28c015e790 2024-04-15T15:16:04.050 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:04.050 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f28a006c8f0 msgr2=0x7f28a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f28a006c8f0 0x7f28a006edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f28a80016e0 tx=0x7f28a8009040 comp rx=0 tx=0).stop 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 msgr2=0x7f28c00ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 0x7f28c00ca190 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f28b800d800 tx=0x7f28b800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 shutdown_connections 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f28a006c8f0 0x7f28a006edb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28c015e790 0x7f28c00ca190 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28c00d3240 0x7f28c00c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 --2- 172.21.15.46:0/4205594010 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f28c00d1440 0x7f28c0152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 >> 172.21.15.46:0/4205594010 conn(0x7f28c008bf60 msgr2=0x7f28c015dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 shutdown_connections 2024-04-15T15:16:04.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.050+0000 7f28c6f9d640 1 -- 172.21.15.46:0/4205594010 wait complete. 2024-04-15T15:16:04.063 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-15T15:16:04.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-15T15:16:04.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-15T15:16:04.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-15T15:16:04.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-15T15:16:04.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-15T15:16:04.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-15T15:16:04.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-15T15:16:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.245+0000 7fb1ccc86640 1 Processor -- start 2024-04-15T15:16:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.245+0000 7fb1ccc86640 1 -- start start 2024-04-15T15:16:04.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.246+0000 7f189d6b8640 1 Processor -- start 2024-04-15T15:16:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1ccc86640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb1c809b0e0 0x7fb1c80cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1ccc86640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c80d0350 con 0x7fb1c809b650 2024-04-15T15:16:04.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1ccc86640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c80ce100 con 0x7fb1c815c8d0 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1ccc86640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb1c80b5b00 con 0x7fb1c809b0e0 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1c6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb1c809b650 0x7fb1c80b5f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55596/0 (socket says 172.21.15.46:55596) 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1c6d9d640 1 -- 172.21.15.46:0/2371876218 learned_addr learned my addr 172.21.15.46:0/2371876218 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1c659c640 1 --1- 172.21.15.46:0/2371876218 >> v1:172.21.15.78:6789/0 conn(0x7fb1c815c8d0 0x7fb1c809a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47104/0 (socket says 172.21.15.46:47104) 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1c5d9b640 1 --2- 172.21.15.46:0/2371876218 >> v2:172.21.15.46:3300/0 conn(0x7fb1c809b0e0 0x7fb1c80cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766630694 0 0) 0x7fb1c80ce100 con 0x7fb1c815c8d0 2024-04-15T15:16:04.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fea2cc9e640 1 Processor -- start 2024-04-15T15:16:04.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1b0007930 con 0x7fb1c815c8d0 2024-04-15T15:16:04.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fea2cc9e640 1 -- start start 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 <== mon.1 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3469314377 0 0) 0x7fb1b0007930 con 0x7fb1c815c8d0 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 >> v2:172.21.15.46:3300/0 conn(0x7fb1c809b0e0 msgr2=0x7fb1c80cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 --2- 172.21.15.46:0/2371876218 >> v2:172.21.15.46:3300/0 conn(0x7fb1c809b0e0 0x7fb1c80cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 >> v1:172.21.15.46:6789/0 conn(0x7fb1c809b650 legacy=0x7fb1c80b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.247+0000 7f189d6b8640 1 -- start start 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7f189d6b8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f18900e7970 0x7f1890053790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7f18977fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1890007180 0x7f1890053cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55610/0 (socket says 172.21.15.46:55610) 2024-04-15T15:16:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7f18977fe640 1 -- 172.21.15.46:0/2616786879 learned_addr learned my addr 172.21.15.46:0/2616786879 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7f9f69d66640 1 Processor -- start 2024-04-15T15:16:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7f9f69d66640 1 -- start start 2024-04-15T15:16:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1890050550 con 0x7f1890007180 2024-04-15T15:16:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18900507f0 con 0x7f18900e7590 2024-04-15T15:16:04.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1890050ad0 con 0x7f18900e7970 2024-04-15T15:16:04.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fede70d6640 1 Processor -- start 2024-04-15T15:16:04.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f18967fc640 1 --2- 172.21.15.46:0/2616786879 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f18900e7970 0x7f1890053790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c80b54b0 con 0x7fb1c815c8d0 2024-04-15T15:16:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fede70d6640 1 -- start start 2024-04-15T15:16:04.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.248+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 <== mon.1 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fb1bc001670 con 0x7fb1c815c8d0 2024-04-15T15:16:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fea2cc9e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 0x7fea280cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f9f69d66640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f640b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f1895ffb640 1 -- 172.21.15.46:0/2616786879 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3251494335 0 0) 0x7f18900507f0 con 0x7f18900e7590 2024-04-15T15:16:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f1895ffb640 1 -- 172.21.15.46:0/2616786879 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1880003520 con 0x7f18900e7590 2024-04-15T15:16:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fea2cc9e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fea280b2f50 0x7fea280ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fea2cc9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea28073590 0x7fea280ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7f9f69d66640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9f640af780 0x7f9f640b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 >> v1:172.21.15.78:6789/0 conn(0x7fb1c815c8d0 legacy=0x7fb1c809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f69d66640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f640b37e0 con 0x7f9f640b4530 2024-04-15T15:16:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f18967fc640 1 -- 172.21.15.46:0/2616786879 >> v1:172.21.15.78:6789/0 conn(0x7f18900e7590 legacy=0x7f1890006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f18967fc640 1 -- 172.21.15.46:0/2616786879 >> v1:172.21.15.46:6789/0 conn(0x7f1890007180 legacy=0x7f1890053cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea2cc9e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fea2806f6e0 con 0x7fea28073590 2024-04-15T15:16:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f18967fc640 1 -- 172.21.15.46:0/2616786879 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1890050db0 con 0x7f18900e7970 2024-04-15T15:16:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f69d66640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f9f640b4990 con 0x7f9f640af780 2024-04-15T15:16:04.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f69d66640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9f640aff60 con 0x7f9f6415b6a0 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7ff3160f7640 1 Processor -- start 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7ff3160f7640 1 -- start start 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea2cc9e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fea28073d10 con 0x7fea280b2f50 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 0x7fea280cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea2cc9e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fea2815ab00 con 0x7fea280b4a10 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 0x7fea280cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42674/0 (socket says 172.21.15.46:42674) 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 -- 172.21.15.46:0/3254481036 learned_addr learned my addr 172.21.15.46:0/3254481036 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fdec759e640 1 Processor -- start 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1b0007dc0 0x7fb1b00081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0008c10 0x7fb1b0009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1b00095d0 0x7fb1b000fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80b5b00 con 0x7fb1b0008c10 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b0010520 con 0x7fb1b0007dc0 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b0010720 con 0x7fb1b00095d0 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c5d9b640 1 --2- 172.21.15.46:0/2371876218 >> v2:172.21.15.46:3300/0 conn(0x7fb1c809b0e0 0x7fb1c80cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c659c640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1b0007dc0 0x7fb1b00081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c5d9b640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0008c10 0x7fb1b0009090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.249+0000 7fb1c6d9d640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1b00095d0 0x7fb1b000fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea26ffd640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.78:3300/0 conn(0x7fea280b2f50 0x7fea280ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea27fff640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.46:3300/0 conn(0x7fea28073590 0x7fea280ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 -- 172.21.15.46:0/3254481036 >> v2:172.21.15.78:3300/0 conn(0x7fea280b2f50 msgr2=0x7fea280ce460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.78:3300/0 conn(0x7fea280b2f50 0x7fea280ce460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 -- 172.21.15.46:0/3254481036 >> v2:172.21.15.46:3300/0 conn(0x7fea28073590 msgr2=0x7fea280ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fdec759e640 1 -- start start 2024-04-15T15:16:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede70d6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fedd80061f0 0x7fedd80066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede70d6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fedd8006be0 0x7fedd803c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7f18967fc640 1 --2- 172.21.15.46:0/2616786879 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f18900e7970 0x7f1890053790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f188c0097d0 tx=0x7f188c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=590947177c10f538 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede70d6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd80e36e0 con 0x7fedd80e3300 2024-04-15T15:16:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede70d6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fedd8057b40 con 0x7fedd80061f0 2024-04-15T15:16:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede70d6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fedd8057e20 con 0x7fedd8006be0 2024-04-15T15:16:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3160f7640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff3080ded90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3160f7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3080df2d0 0x7ff30804f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede68d5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fedd8006be0 0x7fedd803c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede68d5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fedd8006be0 0x7fedd803c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47546/0 (socket says 172.21.15.46:47546) 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fede68d5640 1 -- 172.21.15.46:0/67062811 learned_addr learned my addr 172.21.15.46:0/67062811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3160f7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff308054d40 con 0x7ff30804fc20 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3160f7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff3080507a0 con 0x7ff3080df2d0 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3160f7640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff3080e5da0 con 0x7ff308004a30 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3158f6640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff30804fc20 0x7ff308050090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47128/0 (socket says 172.21.15.46:47128) 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3158f6640 1 -- 172.21.15.46:0/472670729 learned_addr learned my addr 172.21.15.46:0/472670729 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7ff3150f5640 1 --2- 172.21.15.46:0/472670729 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff3080ded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fdec759e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdec00587e0 0x7fdec00ee170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fea277fe640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.46:3300/0 conn(0x7fea28073590 0x7fea280ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fea277fe640 1 -- 172.21.15.46:0/3254481036 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2806f120 con 0x7fea280b4a10 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fea27fff640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.46:3300/0 conn(0x7fea28073590 0x7fea280ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fdec759e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec00e3750 con 0x7fdec00e3370 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fdec759e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec00545e0 con 0x7fdec0006c40 2024-04-15T15:16:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7ff3148f4640 1 --2- 172.21.15.46:0/472670729 >> v2:172.21.15.46:3300/0 conn(0x7ff3080df2d0 0x7ff30804f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fdec759e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fdec0007be0 con 0x7fdec00587e0 2024-04-15T15:16:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fb1c659c640 1 -- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1b00095d0 msgr2=0x7fb1b000fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fb1c659c640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1b00095d0 0x7fb1b000fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fb1c659c640 1 -- 172.21.15.46:0/2371876218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0008c10 msgr2=0x7fb1b0009090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fb1c659c640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0008c10 0x7fb1b0009090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fb1c659c640 1 -- 172.21.15.46:0/2371876218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c80b54b0 con 0x7fb1b0007dc0 2024-04-15T15:16:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7fb1c6d9d640 1 -- 172.21.15.46:0/2371876218 reap_dead start 2024-04-15T15:16:04.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7fb1c659c640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1b0007dc0 0x7fb1b00081e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c190 tx=0x7fb1b0009c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea4a772fd4890ea server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f68d64640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f640b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f69565640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9f640af780 0x7f9f640b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f68d64640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f640b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42664/0 (socket says 172.21.15.46:42664) 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.250+0000 7f9f68d64640 1 -- 172.21.15.46:0/1809613540 learned_addr learned my addr 172.21.15.46:0/1809613540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7f9f69565640 1 -- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 msgr2=0x7f9f640b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7f9f69565640 1 --2- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f640b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7f9f69565640 1 -- 172.21.15.46:0/1809613540 >> v1:172.21.15.46:6789/0 conn(0x7f9f640b4530 legacy=0x7f9f640b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.251+0000 7f9f69565640 1 -- 172.21.15.46:0/1809613540 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f640b01b0 con 0x7f9f640af780 2024-04-15T15:16:04.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fea277fe640 1 --2- 172.21.15.46:0/3254481036 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 0x7fea280cdf20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea280741c0 tx=0x7fea1c02fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84aefb84e54bd571 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc007c40 con 0x7fb1b0007dc0 2024-04-15T15:16:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1bc030070 con 0x7fb1b0007dc0 2024-04-15T15:16:04.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fb1c559a640 1 -- 172.21.15.46:0/2371876218 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc02cd30 con 0x7fb1b0007dc0 2024-04-15T15:16:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fea267fc640 1 -- 172.21.15.46:0/3254481036 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c030840 con 0x7fea280b4a10 2024-04-15T15:16:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7ff3150f5640 1 -- 172.21.15.46:0/472670729 >> v2:172.21.15.46:3300/0 conn(0x7ff3080df2d0 msgr2=0x7ff30804f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7ff3150f5640 1 --2- 172.21.15.46:0/472670729 >> v2:172.21.15.46:3300/0 conn(0x7ff3080df2d0 0x7ff30804f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7ff3150f5640 1 -- 172.21.15.46:0/472670729 >> v1:172.21.15.78:6789/0 conn(0x7ff30804fc20 legacy=0x7ff308050090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7ff3150f5640 1 -- 172.21.15.46:0/472670729 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff308006de0 con 0x7ff308004a30 2024-04-15T15:16:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fea267fc640 1 -- 172.21.15.46:0/3254481036 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1c0309e0 con 0x7fea280b4a10 2024-04-15T15:16:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fea267fc640 1 -- 172.21.15.46:0/3254481036 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c030db0 con 0x7fea280b4a10 2024-04-15T15:16:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fede68d5640 1 -- 172.21.15.46:0/67062811 >> v2:172.21.15.46:3300/0 conn(0x7fedd80061f0 msgr2=0x7fedd80066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fede68d5640 1 --2- 172.21.15.46:0/67062811 >> v2:172.21.15.46:3300/0 conn(0x7fedd80061f0 0x7fedd80066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fede68d5640 1 -- 172.21.15.46:0/67062811 >> v1:172.21.15.46:6789/0 conn(0x7fedd80e3300 legacy=0x7fedd80572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.252+0000 7fede68d5640 1 -- 172.21.15.46:0/67062811 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd8058100 con 0x7fedd8006be0 2024-04-15T15:16:04.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fdec5d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdec0006c40 0x7fdec0053ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47134/0 (socket says 172.21.15.46:47134) 2024-04-15T15:16:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fdec5d9b640 1 -- 172.21.15.46:0/3715696804 learned_addr learned my addr 172.21.15.46:0/3715696804 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7f1895ffb640 1 -- 172.21.15.46:0/2616786879 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c035020 con 0x7f18900e7970 2024-04-15T15:16:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7f1895ffb640 1 -- 172.21.15.46:0/2616786879 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f188c031830 con 0x7f18900e7970 2024-04-15T15:16:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fede68d5640 1 --2- 172.21.15.46:0/67062811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd8006be0 0x7fedd803c120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fede006a8e0 tx=0x7fede00a7160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e22cdf6a574f8cc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fdec6d9d640 1 --2- 172.21.15.46:0/3715696804 >> v2:172.21.15.78:3300/0 conn(0x7fdec00587e0 0x7fdec00ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 -- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1b0007dc0 msgr2=0x7fb1b00081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1b0007dc0 0x7fb1b00081e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c190 tx=0x7fb1b0009c40 comp rx=0 tx=0).stop 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 -- 172.21.15.46:0/2371876218 shutdown_connections 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 --2- 172.21.15.46:0/2371876218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1b0007dc0 0x7fb1b00081e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 -- 172.21.15.46:0/2371876218 >> 172.21.15.46:0/2371876218 conn(0x7fb1c80b3fb0 msgr2=0x7fb1c80b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 -- 172.21.15.46:0/3254481036 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 msgr2=0x7fea280cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/3254481036 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 0x7fea280cdf20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fea280741c0 tx=0x7fea1c02fca0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 -- 172.21.15.46:0/3254481036 shutdown_connections 2024-04-15T15:16:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.46:3300/0 conn(0x7fea28073590 0x7fea280ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/3254481036 >> v2:172.21.15.78:3300/0 conn(0x7fea280b2f50 0x7fea280ce460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/3254481036 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea280b4a10 0x7fea280cdf20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fea2cc9e640 1 -- 172.21.15.46:0/3254481036 >> 172.21.15.46:0/3254481036 conn(0x7fea280b1c50 msgr2=0x7fea280b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 -- 172.21.15.46:0/2371876218 shutdown_connections 2024-04-15T15:16:04.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fb1ccc86640 1 -- 172.21.15.46:0/2371876218 wait complete. 2024-04-15T15:16:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7ff3150f5640 1 --2- 172.21.15.46:0/472670729 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff3080ded90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0097a0 tx=0x7ff30c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f0a8f20004c7965 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fdec6d9d640 1 -- 172.21.15.46:0/3715696804 >> v1:172.21.15.78:6789/0 conn(0x7fdec0006c40 legacy=0x7fdec0053ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fdec6d9d640 1 -- 172.21.15.46:0/3715696804 >> v1:172.21.15.46:6789/0 conn(0x7fdec00e3370 legacy=0x7fdec00537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fdec6d9d640 1 -- 172.21.15.46:0/3715696804 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00070a0 con 0x7fdec00587e0 2024-04-15T15:16:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fea2cc9e640 1 -- 172.21.15.46:0/3254481036 shutdown_connections 2024-04-15T15:16:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fea2cc9e640 1 -- 172.21.15.46:0/3254481036 wait complete. 2024-04-15T15:16:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f1895ffb640 1 -- 172.21.15.46:0/2616786879 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c031c40 con 0x7f18900e7970 2024-04-15T15:16:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f18900e7970 msgr2=0x7f1890053790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f189d6b8640 1 --2- 172.21.15.46:0/2616786879 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f18900e7970 0x7f1890053790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f188c0097d0 tx=0x7f188c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fede50d2640 1 -- 172.21.15.46:0/67062811 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede00b4070 con 0x7fedd8006be0 2024-04-15T15:16:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fede50d2640 1 -- 172.21.15.46:0/67062811 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fede00af440 con 0x7fedd8006be0 2024-04-15T15:16:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.253+0000 7fede50d2640 1 -- 172.21.15.46:0/67062811 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede00af8b0 con 0x7fedd8006be0 2024-04-15T15:16:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f9f69565640 1 --2- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640af780 0x7f9f640b13a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009d90 tx=0x7f9f5c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e2858ebfd2c3175 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 -- 172.21.15.46:0/67062811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd8006be0 msgr2=0x7fedd803c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 --2- 172.21.15.46:0/67062811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd8006be0 0x7fedd803c120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fede006a8e0 tx=0x7fede00a7160 comp rx=0 tx=0).stop 2024-04-15T15:16:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 shutdown_connections 2024-04-15T15:16:04.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f189d6b8640 1 --2- 172.21.15.46:0/2616786879 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f18900e7970 0x7f1890053790 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 >> 172.21.15.46:0/2616786879 conn(0x7f1890055bc0 msgr2=0x7f1890055fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7ff307fff640 1 -- 172.21.15.46:0/472670729 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c034070 con 0x7ff308004a30 2024-04-15T15:16:04.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7ff307fff640 1 -- 172.21.15.46:0/472670729 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30c030800 con 0x7ff308004a30 2024-04-15T15:16:04.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fdec6d9d640 1 --2- 172.21.15.46:0/3715696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00587e0 0x7fdec00ee170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdec8051700 tx=0x7fdec80a7210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62898198d914bd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7ff307fff640 1 -- 172.21.15.46:0/472670729 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30c030bd0 con 0x7ff308004a30 2024-04-15T15:16:04.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f9f637fe640 1 -- 172.21.15.46:0/1809613540 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c039070 con 0x7f9f640af780 2024-04-15T15:16:04.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 -- 172.21.15.46:0/67062811 shutdown_connections 2024-04-15T15:16:04.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 --2- 172.21.15.46:0/67062811 >> v2:172.21.15.46:3300/0 conn(0x7fedd80061f0 0x7fedd80066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 --2- 172.21.15.46:0/67062811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd8006be0 0x7fedd803c120 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 -- 172.21.15.46:0/67062811 >> 172.21.15.46:0/67062811 conn(0x7fedd8058ef0 msgr2=0x7fedd8059300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f9f637fe640 1 -- 172.21.15.46:0/1809613540 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f5c02cc50 con 0x7f9f640af780 2024-04-15T15:16:04.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fdec559a640 1 -- 172.21.15.46:0/3715696804 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec80b4070 con 0x7fdec00587e0 2024-04-15T15:16:04.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7f9f637fe640 1 -- 172.21.15.46:0/1809613540 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c034440 con 0x7f9f640af780 2024-04-15T15:16:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fdec559a640 1 -- 172.21.15.46:0/3715696804 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec80af640 con 0x7fdec00587e0 2024-04-15T15:16:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fdec559a640 1 -- 172.21.15.46:0/3715696804 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec80afab0 con 0x7fdec00587e0 2024-04-15T15:16:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 -- 172.21.15.46:0/67062811 shutdown_connections 2024-04-15T15:16:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.254+0000 7fede70d6640 1 -- 172.21.15.46:0/67062811 wait complete. 2024-04-15T15:16:04.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 shutdown_connections 2024-04-15T15:16:04.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7ff3160f7640 1 -- 172.21.15.46:0/472670729 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 msgr2=0x7ff3080ded90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7ff3160f7640 1 --2- 172.21.15.46:0/472670729 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff3080ded90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0097a0 tx=0x7ff30c02b940 comp rx=0 tx=0).stop 2024-04-15T15:16:04.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7f9f69d66640 1 -- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640af780 msgr2=0x7f9f640b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7fdec759e640 1 -- 172.21.15.46:0/3715696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00587e0 msgr2=0x7fdec00ee170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7fdec759e640 1 --2- 172.21.15.46:0/3715696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00587e0 0x7fdec00ee170 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fdec8051700 tx=0x7fdec80a7210 comp rx=0 tx=0).stop 2024-04-15T15:16:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7ff3160f7640 1 -- 172.21.15.46:0/472670729 shutdown_connections 2024-04-15T15:16:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7ff3160f7640 1 --2- 172.21.15.46:0/472670729 >> v2:172.21.15.46:3300/0 conn(0x7ff3080df2d0 0x7ff30804f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7ff3160f7640 1 --2- 172.21.15.46:0/472670729 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff3080ded90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7ff3160f7640 1 -- 172.21.15.46:0/472670729 >> 172.21.15.46:0/472670729 conn(0x7ff30800a400 msgr2=0x7ff30800a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7f9f69d66640 1 --2- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640af780 0x7f9f640b13a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009d90 tx=0x7f9f5c02bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fdec759e640 1 -- 172.21.15.46:0/3715696804 shutdown_connections 2024-04-15T15:16:04.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fea2cc9e640 1 Processor -- start 2024-04-15T15:16:04.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7f9f69d66640 1 -- 172.21.15.46:0/1809613540 shutdown_connections 2024-04-15T15:16:04.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7f9f69d66640 1 --2- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640af780 0x7f9f640b13a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7f9f69d66640 1 --2- 172.21.15.46:0/1809613540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f640b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7f9f69d66640 1 -- 172.21.15.46:0/1809613540 >> 172.21.15.46:0/1809613540 conn(0x7f9f64073060 msgr2=0x7f9f640b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fdec759e640 1 --2- 172.21.15.46:0/3715696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00587e0 0x7fdec00ee170 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7ff3160f7640 1 -- 172.21.15.46:0/472670729 shutdown_connections 2024-04-15T15:16:04.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7ff3160f7640 1 -- 172.21.15.46:0/472670729 wait complete. 2024-04-15T15:16:04.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fdec759e640 1 -- 172.21.15.46:0/3715696804 >> 172.21.15.46:0/3715696804 conn(0x7fdec003aa50 msgr2=0x7fdec003ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fea2cc9e640 1 -- start start 2024-04-15T15:16:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fdec759e640 1 -- 172.21.15.46:0/3715696804 shutdown_connections 2024-04-15T15:16:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fdec759e640 1 -- 172.21.15.46:0/3715696804 wait complete. 2024-04-15T15:16:04.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fb1ccc86640 1 Processor -- start 2024-04-15T15:16:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.255+0000 7f189d6b8640 1 -- 172.21.15.46:0/2616786879 wait complete. 2024-04-15T15:16:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fede70d6640 1 Processor -- start 2024-04-15T15:16:04.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fede70d6640 1 -- start start 2024-04-15T15:16:04.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fe51e900640 1 Processor -- start 2024-04-15T15:16:04.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.256+0000 7fe51e900640 1 -- start start 2024-04-15T15:16:04.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.257+0000 7f9f69d66640 1 -- 172.21.15.46:0/1809613540 shutdown_connections 2024-04-15T15:16:04.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.257+0000 7f9f69d66640 1 -- 172.21.15.46:0/1809613540 wait complete. 2024-04-15T15:16:04.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.257+0000 7f189d6b8640 1 Processor -- start 2024-04-15T15:16:04.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.257+0000 7f189d6b8640 1 -- start start 2024-04-15T15:16:04.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.257+0000 7fea2cc9e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea28073590 0x7fea2809ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fea2cc9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea280b2f50 0x7fea2814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fede70d6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd80061f0 0x7fedd80d2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fede70d6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd8006be0 0x7fedd8038800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51e900640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe51815eac0 0x7fe518070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fea277fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea28073590 0x7fea2809ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51e900640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51809b710 0x7fe5180d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51e900640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51809bc10 con 0x7fe51809a9d0 2024-04-15T15:16:04.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51e900640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe5180706c0 con 0x7fe51815eac0 2024-04-15T15:16:04.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51e900640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe5180b1920 con 0x7fe51809b710 2024-04-15T15:16:04.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f189d6b8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1890007180 0x7f1890053d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f189d6b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18900e7590 0x7f1890054280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f189d6b8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1890048d80 0x7f189004dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fea277fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea28073590 0x7fea2809ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42692/0 (socket says 172.21.15.46:42692) 2024-04-15T15:16:04.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f189d6b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f18900515b0 con 0x7f18900e7590 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51ce9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe51809a9d0 0x7fe5180cca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47144/0 (socket says 172.21.15.46:47144) 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe51ce9d640 1 -- 172.21.15.46:0/610810160 learned_addr learned my addr 172.21.15.46:0/610810160 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f189d6b8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1890055800 con 0x7f1890048d80 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f18977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1890048d80 0x7f189004dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f18977fe640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1890048d80 0x7f189004dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47596/0 (socket says 172.21.15.46:47596) 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f18977fe640 1 -- 172.21.15.46:0/572255455 learned_addr learned my addr 172.21.15.46:0/572255455 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7f189d6b8640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f18900495e0 con 0x7f1890007180 2024-04-15T15:16:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe517fff640 1 --2- 172.21.15.46:0/610810160 >> v2:172.21.15.78:3300/0 conn(0x7fe51815eac0 0x7fe518070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fe5177fe640 1 --2- 172.21.15.46:0/610810160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51809b710 0x7fe5180d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea280b2f50 0x7fea2814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea280b2f50 0x7fea2814e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42128/0 (socket says 172.21.15.46:42128) 2024-04-15T15:16:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea2cc9e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea280b4a10 0x7fea28156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.257+0000 7fb1ccc86640 1 -- start start 2024-04-15T15:16:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1ccc86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0007dc0 0x7fb1c815d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1ccc86640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c815ea00 0x7fb1c81589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1ccc86640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1c80ca8b0 0x7fb1c80cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1ccc86640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80cdbf0 con 0x7fb1b0007dc0 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1ccc86640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80cda70 con 0x7fb1c815ea00 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1ccc86640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c80cdd70 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0007dc0 0x7fb1c815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0007dc0 0x7fb1c815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42148/0 (socket says 172.21.15.46:42148) 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea2cc9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fea280ccb30 con 0x7fea280b2f50 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea2cc9e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea280cc850 con 0x7fea280b4a10 2024-04-15T15:16:04.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea2cc9e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fea280cce10 con 0x7fea28073590 2024-04-15T15:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1c659c640 1 -- 172.21.15.46:0/3381314023 learned_addr learned my addr 172.21.15.46:0/3381314023 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1c5d9b640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c815ea00 0x7fb1c81589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fb1c6d9d640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1c80ca8b0 0x7fb1c80cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7ff3160f7640 1 Processor -- start 2024-04-15T15:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7ff3160f7640 1 -- start start 2024-04-15T15:16:04.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3160f7640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff308047710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3160f7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30804fc20 0x7ff308034930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3160f7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3080df2d0 0x7ff30803ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3160f7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff308007680 con 0x7ff30804fc20 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3160f7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff3080073a0 con 0x7ff3080df2d0 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3160f7640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff308007ba0 con 0x7ff308004a30 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f1896ffd640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1890007180 0x7f1890053d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea26ffd640 1 -- 172.21.15.46:0/2589717156 learned_addr learned my addr 172.21.15.46:0/2589717156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea27fff640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea280b4a10 0x7fea28156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fb1c6d9d640 1 -- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c815ea00 msgr2=0x7fb1c81589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fb1c6d9d640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1c815ea00 0x7fb1c81589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fb1c6d9d640 1 -- 172.21.15.46:0/3381314023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0007dc0 msgr2=0x7fb1c815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fb1c6d9d640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b0007dc0 0x7fb1c815d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fb1c6d9d640 1 -- 172.21.15.46:0/3381314023 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c815dd20 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f18977fe640 1 -- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1890007180 msgr2=0x7f1890053d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f18977fe640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1890007180 0x7f1890053d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f18977fe640 1 -- 172.21.15.46:0/572255455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18900e7590 msgr2=0x7f1890054280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f18977fe640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18900e7590 0x7f1890054280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f18977fe640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18900498c0 con 0x7f1890048d80 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3148f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30804fc20 0x7ff308034930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fede58d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd8006be0 0x7fedd8038800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3148f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30804fc20 0x7ff308034930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42156/0 (socket says 172.21.15.46:42156) 2024-04-15T15:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3148f4640 1 -- 172.21.15.46:0/1455474953 learned_addr learned my addr 172.21.15.46:0/1455474953 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f9f69d66640 1 Processor -- start 2024-04-15T15:16:04.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fede70d6640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fedd80e3300 0x7fedd803a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fede70d6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8058be0 con 0x7fedd8006be0 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fede70d6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8006fc0 con 0x7fedd80061f0 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fede70d6640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fedd80d2860 con 0x7fedd80e3300 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fede58d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd8006be0 0x7fedd8038800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42136/0 (socket says 172.21.15.46:42136) 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd80061f0 0x7fedd80d2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede58d3640 1 -- 172.21.15.46:0/193467731 learned_addr learned my addr 172.21.15.46:0/193467731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7f9f69d66640 1 -- start start 2024-04-15T15:16:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fb1c6d9d640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1c80ca8b0 0x7fb1c80cad40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80687d0 tx=0x7fb1c000d810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7ff3158f6640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3080df2d0 0x7ff30803ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3150f5640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff308047710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea26ffd640 1 -- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea28073590 msgr2=0x7fea2809ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fea26ffd640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea28073590 0x7fea2809ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea26ffd640 1 -- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea280b4a10 msgr2=0x7fea28156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea26ffd640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea280b4a10 0x7fea28156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea26ffd640 1 -- 172.21.15.46:0/2589717156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2809b050 con 0x7fea280b2f50 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fe5177fe640 1 -- 172.21.15.46:0/610810160 >> v2:172.21.15.78:3300/0 conn(0x7fe51815eac0 msgr2=0x7fe518070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fe5177fe640 1 --2- 172.21.15.46:0/610810160 >> v2:172.21.15.78:3300/0 conn(0x7fe51815eac0 0x7fe518070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fe5177fe640 1 -- 172.21.15.46:0/610810160 >> v1:172.21.15.78:6789/0 conn(0x7fe51809a9d0 legacy=0x7fe5180cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fe5177fe640 1 -- 172.21.15.46:0/610810160 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5180d0960 con 0x7fe51809b710 2024-04-15T15:16:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3158f6640 1 -- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 msgr2=0x7ff308047710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3158f6640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff308004a30 0x7ff308047710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3158f6640 1 -- 172.21.15.46:0/1455474953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30804fc20 msgr2=0x7ff308034930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3158f6640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff30804fc20 0x7ff308034930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3158f6640 1 -- 172.21.15.46:0/1455474953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3080377a0 con 0x7ff3080df2d0 2024-04-15T15:16:04.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea26ffd640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea280b2f50 0x7fea2814e0d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea1800d620 tx=0x7fea1800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0004d70 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fdec759e640 1 Processor -- start 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.258+0000 7fdec759e640 1 -- start start 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.259+0000 7fdec759e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec0006c40 0x7fdec01856d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00587e0 0x7fdec00e51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec759e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdec00e3370 0x7fdec00dc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec759e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0007900 con 0x7fdec00587e0 2024-04-15T15:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec759e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0007620 con 0x7fdec0006c40 2024-04-15T15:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec759e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00572b0 con 0x7fdec00e3370 2024-04-15T15:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00587e0 0x7fdec00e51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00587e0 0x7fdec00e51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42166/0 (socket says 172.21.15.46:42166) 2024-04-15T15:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fdec5d9b640 1 -- 172.21.15.46:0/3170196910 learned_addr learned my addr 172.21.15.46:0/3170196910 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff3158f6640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3080df2d0 0x7ff30803ad10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff310051650 tx=0x7ff3100694c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c815e000 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c815e5c0 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff310069f10 con 0x7ff3080df2d0 2024-04-15T15:16:04.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7f18977fe640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1890048d80 0x7f189004dce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1898069090 tx=0x7f18980694c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1800b620 con 0x7fea280b2f50 2024-04-15T15:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea18019070 con 0x7fea280b2f50 2024-04-15T15:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1800ba00 con 0x7fea280b2f50 2024-04-15T15:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff310090020 con 0x7ff3080df2d0 2024-04-15T15:16:04.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea281e1d40 con 0x7fea280b2f50 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea281e2300 con 0x7fea280b2f50 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede68d5640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fedd80e3300 0x7fedd803a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 -- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fedd80e3300 msgr2=0x7fedd803a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fedd80e3300 0x7fedd803a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 -- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd8006be0 msgr2=0x7fedd8038800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd8006be0 0x7fedd8038800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 -- 172.21.15.46:0/193467731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd80e0ce0 con 0x7fedd80061f0 2024-04-15T15:16:04.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.260+0000 7fede60d4640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd80061f0 0x7fedd80d2320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fedd000b3e0 tx=0x7fedd000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0015020 con 0x7fedd80061f0 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedd0011830 con 0x7fedd80061f0 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec6d9d640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdec00e3370 0x7fdec00dc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec5d9b640 1 -- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdec00e3370 msgr2=0x7fdec00dc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec5d9b640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdec00e3370 0x7fdec00dc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec5d9b640 1 -- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec0006c40 msgr2=0x7fdec01856d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec5d9b640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec0006c40 0x7fdec01856d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec5d9b640 1 -- 172.21.15.46:0/3170196910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00df610 con 0x7fdec00587e0 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fdec5d9b640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00587e0 0x7fdec00e51f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdebc00c460 tx=0x7fdebc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7f9f69d66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f640af780 0x7f9f64145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7f9f69d66640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640b4530 0x7f9f64146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7f9f69d66640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f64155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7f9f63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640b4530 0x7f9f64146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7f9f63fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640b4530 0x7f9f64146360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47624/0 (socket says 172.21.15.46:47624) 2024-04-15T15:16:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f9f63fff640 1 -- 172.21.15.46:0/824937446 learned_addr learned my addr 172.21.15.46:0/824937446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fe5177fe640 1 --2- 172.21.15.46:0/610810160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51809b710 0x7fe5180d1440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5080097a0 tx=0x7fe50802b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f74e05746bbe55 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fe516ffd640 1 -- 172.21.15.46:0/610810160 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508037070 con 0x7fe51809b710 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fe516ffd640 1 -- 172.21.15.46:0/610810160 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe508030710 con 0x7fe51809b710 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1c00123d0 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c00070d0 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.261+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31006bc60 con 0x7ff3080df2d0 2024-04-15T15:16:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f9f69565640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f64155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fe516ffd640 1 -- 172.21.15.46:0/610810160 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50802be60 con 0x7fe51809b710 2024-04-15T15:16:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f9f68d64640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f640af780 0x7f9f64145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6409ab90 con 0x7f9f640af780 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1898069f10 con 0x7f1890048d80 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1898090020 con 0x7f1890048d80 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f189d6b8640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1890049ba0 con 0x7f1890048d80 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9f640b0800 con 0x7f9f640b4530 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0011c00 con 0x7fedd80061f0 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9f6409ad10 con 0x7f9f6415b6a0 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7f189d6b8640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1890038210 con 0x7f1890048d80 2024-04-15T15:16:04.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fea2804fd50 con 0x7fea280b2f50 2024-04-15T15:16:04.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fede70d6640 1 -- 172.21.15.46:0/193467731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd80e0fc0 con 0x7fedd80061f0 2024-04-15T15:16:04.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fede70d6640 1 -- 172.21.15.46:0/193467731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd80e1580 con 0x7fedd80061f0 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f69565640 1 -- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640b4530 msgr2=0x7f9f64146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f69565640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f640b4530 0x7f9f64146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f69565640 1 -- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f640af780 msgr2=0x7f9f64145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f69565640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f640af780 0x7f9f64145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f69565640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f6415d580 con 0x7f9f6415b6a0 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 -- 172.21.15.46:0/610810160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51809b710 msgr2=0x7fe5180d1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 --2- 172.21.15.46:0/610810160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51809b710 0x7fe5180d1440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe5080097a0 tx=0x7fe50802b660 comp rx=0 tx=0).stop 2024-04-15T15:16:04.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 -- 172.21.15.46:0/610810160 shutdown_connections 2024-04-15T15:16:04.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189806bc60 con 0x7f1890048d80 2024-04-15T15:16:04.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 --2- 172.21.15.46:0/610810160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51809b710 0x7fe5180d1440 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30804eb20 con 0x7ff3080df2d0 2024-04-15T15:16:04.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff308034e70 con 0x7ff3080df2d0 2024-04-15T15:16:04.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 --2- 172.21.15.46:0/610810160 >> v2:172.21.15.78:3300/0 conn(0x7fe51815eac0 0x7fe518070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f69565640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f64155100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c033000 tx=0x7f9f5c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f189d6b8640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1878001030 con 0x7f1890048d80 2024-04-15T15:16:04.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 -- 172.21.15.46:0/610810160 >> 172.21.15.46:0/610810160 conn(0x7fe5180b3ec0 msgr2=0x7fe5180b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.262+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1ac001030 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb1c0007270 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 -- 172.21.15.46:0/610810160 shutdown_connections 2024-04-15T15:16:04.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fe51e900640 1 -- 172.21.15.46:0/610810160 wait complete. 2024-04-15T15:16:04.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fedbc001030 con 0x7fedd80061f0 2024-04-15T15:16:04.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.263+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c044020 con 0x7f9f6415b6a0 2024-04-15T15:16:04.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f5c02ca50 con 0x7f9f6415b6a0 2024-04-15T15:16:04.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c039040 con 0x7f9f6415b6a0 2024-04-15T15:16:04.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f6415cfc0 con 0x7f9f6415b6a0 2024-04-15T15:16:04.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f6415d8e0 con 0x7f9f6415b6a0 2024-04-15T15:16:04.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7fb1b6ffd640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb1a806c9c0 0x7fb1a806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea1800bba0 con 0x7fea280b2f50 2024-04-15T15:16:04.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.264+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb1c008bde0 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fb1b6ffd640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb1a80760c0 0x7fb1a8078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f189806d110 con 0x7f1890048d80 2024-04-15T15:16:04.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 --> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1c0014e70 con 0x7fb1a80760c0 2024-04-15T15:16:04.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff308005710 con 0x7ff3080df2d0 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb1c0059250 con 0x7fb1c80ca8b0 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fea0ffff640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea1406c940 0x7fea1406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fea1808cd20 con 0x7fea280b2f50 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fea0ffff640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7fea140760c0 0x7fea14078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 --> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea1801aec0 con 0x7fea140760c0 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fea1808d0d0 con 0x7fea280b2f50 2024-04-15T15:16:04.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fedd0018020 con 0x7fedd80061f0 2024-04-15T15:16:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc005ce0 con 0x7fdec00587e0 2024-04-15T15:16:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdebc007070 con 0x7fdec00587e0 2024-04-15T15:16:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc007440 con 0x7fdec00587e0 2024-04-15T15:16:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fb1c5d9b640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb1a80760c0 0x7fb1a8078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fea27fff640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7fea140760c0 0x7fea14078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7fe51e900640 1 Processor -- start 2024-04-15T15:16:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7f187f7fe640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f188406c8f0 0x7f188406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fb1c659c640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb1a806c9c0 0x7fb1a806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec00df8f0 con 0x7fdec00587e0 2024-04-15T15:16:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fea277fe640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea1406c940 0x7fea1406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fea277fe640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea1406c940 0x7fea1406ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea1c039000 tx=0x7fea1c034000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fedd6ffd640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fedc406c8f0 0x7fedc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fede58d3640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fedc406c8f0 0x7fedc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fedd008c3a0 con 0x7fedd80061f0 2024-04-15T15:16:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec0185c10 con 0x7fdec00587e0 2024-04-15T15:16:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.265+0000 7f9f4effd640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f3c001030 con 0x7f9f6415b6a0 2024-04-15T15:16:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f5c047020 con 0x7f9f6415b6a0 2024-04-15T15:16:04.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fe51e900640 1 -- start start 2024-04-15T15:16:04.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7fb1c5d9b640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb1a80760c0 0x7fb1a8078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff31006d110 con 0x7ff3080df2d0 2024-04-15T15:16:04.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7ff305ffb640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f406c9c0 0x7ff2f406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff310106930 con 0x7ff3080df2d0 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7ff305ffb640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7ff2f4076140 0x7ff2f4078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 --> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2f4078c50 con 0x7ff2f4076140 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff31010b2a0 con 0x7ff3080df2d0 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f18981072e0 con 0x7f1890048d80 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7f187f7fe640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f1884076060 0x7f18840784a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7f1896ffd640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f188406c8f0 0x7f188406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1884078b70 con 0x7f1884076060 2024-04-15T15:16:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.266+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f18981076d0 con 0x7f1890048d80 2024-04-15T15:16:04.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fedd6ffd640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fedc4076070 0x7fedc40784b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 --> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fedd0011830 con 0x7fedc4076070 2024-04-15T15:16:04.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7f18967fc640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f1884076060 0x7f18840784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fede58d3640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fedc406c8f0 0x7fedc406edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fedd8038450 tx=0x7feddc006f90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fedd008c790 con 0x7fedd80061f0 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51e900640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe51809a9d0 0x7fe5180b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51e900640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe51809b710 0x7fe518158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51e900640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51815eac0 0x7fe5180b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51e900640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe5180d1980 con 0x7fe51809b710 2024-04-15T15:16:04.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fede68d5640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fedc4076070 0x7fedc40784b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7f9f617fa640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9f4806c940 0x7f9f4806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51e900640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe5180d0fb0 con 0x7fe51809a9d0 2024-04-15T15:16:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51e900640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe51815eea0 con 0x7fe51815eac0 2024-04-15T15:16:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe517fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe51809a9d0 0x7fe5180b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== osd.5 v2:172.21.15.78:6816/2916087727 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb1c0014e70 con 0x7fb1a80760c0 2024-04-15T15:16:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe517fff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe51809a9d0 0x7fe5180b5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47640/0 (socket says 172.21.15.46:47640) 2024-04-15T15:16:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9f5c0ae360 con 0x7f9f6415b6a0 2024-04-15T15:16:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe517fff640 1 -- 172.21.15.46:0/982608088 learned_addr learned my addr 172.21.15.46:0/982608088 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe5177fe640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe51809b710 0x7fe518158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.267+0000 7fe51ce9d640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51815eac0 0x7fe5180b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fea27fff640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7fea140760c0 0x7fea14078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f9f617fa640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f9f48076040 0x7f9f48078480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7ff3148f4640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7ff2f4076140 0x7ff2f4078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f5c02ee50 con 0x7f9f48076040 2024-04-15T15:16:04.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7ff3150f5640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f406c9c0 0x7ff2f406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fede68d5640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fedc4076070 0x7fedc40784b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdeac001030 con 0x7fdec00587e0 2024-04-15T15:16:04.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9f5c0b5050 con 0x7f9f6415b6a0 2024-04-15T15:16:04.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f9f68d64640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9f4806c940 0x7f9f4806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f9f63fff640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f9f48076040 0x7f9f48078480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe517fff640 1 -- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51815eac0 msgr2=0x7fe5180b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe517fff640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe51815eac0 0x7fe5180b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe517fff640 1 -- 172.21.15.46:0/982608088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe51809b710 msgr2=0x7fe518158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe517fff640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe51809b710 0x7fe518158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe517fff640 1 -- 172.21.15.46:0/982608088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5180b6150 con 0x7fe51809a9d0 2024-04-15T15:16:04.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fb1c659c640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb1a806c9c0 0x7fb1a806ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc007e90 tx=0x7fb1bc02f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe517fff640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe51809a9d0 0x7fe5180b5b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00e9b0 tx=0x7fe50c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f1896ffd640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f188406c8f0 0x7f188406edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f18880011e0 tx=0x7f1888009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c017070 con 0x7fe51809a9d0 2024-04-15T15:16:04.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f18967fc640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f1884076060 0x7f18840784a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe50c00c9a0 con 0x7fe51809a9d0 2024-04-15T15:16:04.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c00cd90 con 0x7fe51809a9d0 2024-04-15T15:16:04.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.268+0000 7f9f68d64640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9f4806c940 0x7f9f4806ee00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009bd0 tx=0x7f9f58009240 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7f9f63fff640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f9f48076040 0x7f9f48078480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7fe51e900640 1 -- 172.21.15.46:0/982608088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5180b6430 con 0x7fe51809a9d0 2024-04-15T15:16:04.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7ff3148f4640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7ff2f4076140 0x7ff2f4078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== osd.1 v2:172.21.15.46:6817/1258448605 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fedd0011830 con 0x7fedc4076070 2024-04-15T15:16:04.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdebc0075e0 con 0x7fdec00587e0 2024-04-15T15:16:04.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7fe51e900640 1 -- 172.21.15.46:0/982608088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5180b69f0 con 0x7fe51809a9d0 2024-04-15T15:16:04.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== osd.7 v2:172.21.15.78:6808/563779653 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1884078b70 con 0x7f1884076060 2024-04-15T15:16:04.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== osd.3 v2:172.21.15.46:6808/3958614464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f5c02ee50 con 0x7f9f48076040 2024-04-15T15:16:04.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.269+0000 7ff3150f5640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f406c9c0 0x7ff2f406ee80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff30c036000 tx=0x7ff30c03b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== osd.6 v2:172.21.15.78:6817/3189427077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2f4078c50 con 0x7ff2f4076140 2024-04-15T15:16:04.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fdeb6ffd640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdea806c940 0x7fdea806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== osd.0 v2:172.21.15.46:6816/2271822553 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea1801aec0 con 0x7fea140760c0 2024-04-15T15:16:04.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdebc08bac0 con 0x7fdec00587e0 2024-04-15T15:16:04.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fdec659c640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdea806c940 0x7fdea806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fdeb6ffd640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fdea8076040 0x7fdea8078480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 --> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdea8078b50 con 0x7fdea8076040 2024-04-15T15:16:04.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.270+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdebc059410 con 0x7fdec00587e0 2024-04-15T15:16:04.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.271+0000 7fe51e900640 1 -- 172.21.15.46:0/982608088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5180c79d0 con 0x7fe51809a9d0 2024-04-15T15:16:04.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.271+0000 7fdec6d9d640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fdea8076040 0x7fdea8078480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.271+0000 7fdec659c640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdea806c940 0x7fdea806ee00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000f60 tx=0x7fdeb80079b0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.273+0000 7fdec6d9d640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fdea8076040 0x7fdea8078480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.273+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe50c01a070 con 0x7fe51809a9d0 2024-04-15T15:16:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.273+0000 7fe514ff9640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe50406c8f0 0x7fe50406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.273+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe50c08c2a0 con 0x7fe51809a9d0 2024-04-15T15:16:04.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.274+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== osd.2 v2:172.21.15.46:6800/1733872723 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdea8078b50 con 0x7fdea8076040 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.274+0000 7fe514ff9640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7fe504076070 0x7fe5040784b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.274+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 --> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe504078b80 con 0x7fe504076070 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.274+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe50c093050 con 0x7fe51809a9d0 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.275+0000 7fe51ce9d640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7fe504076070 0x7fe5040784b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.275+0000 7fe5177fe640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe50406c8f0 0x7fe50406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.276+0000 7fe51ce9d640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7fe504076070 0x7fe5040784b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.276+0000 7fe5177fe640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe50406c8f0 0x7fe50406edb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe508034000 tx=0x7fe508008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.276+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== osd.4 v2:172.21.15.78:6800/3745610613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe504078b80 con 0x7fe504076070 2024-04-15T15:16:04.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.298+0000 7f9f4effd640 1 -- 172.21.15.46:0/824937446 --> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9f3c002dd0 con 0x7f9f48076040 2024-04-15T15:16:04.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.299+0000 7f9f617fa640 1 -- 172.21.15.46:0/824937446 <== osd.3 v2:172.21.15.46:6808/3958614464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9f3c002dd0 con 0x7f9f48076040 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.299+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f9f48076040 msgr2=0x7f9f48078480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.299+0000 7f9f69d66640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f9f48076040 0x7f9f48078480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.299+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9f4806c940 msgr2=0x7f9f4806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.299+0000 7f9f69d66640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9f4806c940 0x7f9f4806ee00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009bd0 tx=0x7f9f58009240 comp rx=0 tx=0).stop 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 msgr2=0x7f9f64155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69d66640 1 --2- 172.21.15.46:0/824937446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9f6415b6a0 0x7f9f64155100 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c033000 tx=0x7f9f5c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69565640 1 -- 172.21.15.46:0/824937446 reap_dead start 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 shutdown_connections 2024-04-15T15:16:04.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 >> 172.21.15.46:0/824937446 conn(0x7f9f64073060 msgr2=0x7f9f64158d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 shutdown_connections 2024-04-15T15:16:04.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.300+0000 7f9f69d66640 1 -- 172.21.15.46:0/824937446 wait complete. 2024-04-15T15:16:04.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.302+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 --> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fedbc002dd0 con 0x7fedc4076070 2024-04-15T15:16:04.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.302+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 --> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff3080e3d70 con 0x7ff2f4076140 2024-04-15T15:16:04.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.303+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 --> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb1ac002dd0 con 0x7fb1a80760c0 2024-04-15T15:16:04.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.303+0000 7fedd6ffd640 1 -- 172.21.15.46:0/193467731 <== osd.1 v2:172.21.15.46:6817/1258448605 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fedbc002dd0 con 0x7fedc4076070 2024-04-15T15:16:04.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.303+0000 7ff305ffb640 1 -- 172.21.15.46:0/1455474953 <== osd.6 v2:172.21.15.78:6817/3189427077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff3080e3d70 con 0x7ff2f4076140 2024-04-15T15:16:04.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.303+0000 7fb1b6ffd640 1 -- 172.21.15.46:0/3381314023 <== osd.5 v2:172.21.15.78:6816/2916087727 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb1ac002dd0 con 0x7fb1a80760c0 2024-04-15T15:16:04.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.303+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fedc4076070 msgr2=0x7fedc40784b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.303+0000 7fedd4ff9640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fedc4076070 0x7fedc40784b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb1a80760c0 msgr2=0x7fb1a8078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fb1ccc86640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb1a80760c0 0x7fb1a8078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7ff2f4076140 msgr2=0x7ff2f4078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7f189d6b8640 1 -- 172.21.15.46:0/572255455 --> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1878002dd0 con 0x7f1884076060 2024-04-15T15:16:04.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7ff2f4076140 0x7ff2f4078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fedc406c8f0 msgr2=0x7fedc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fedd4ff9640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fedc406c8f0 0x7fedc406edb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fedd8038450 tx=0x7feddc006f90 comp rx=0 tx=0).stop 2024-04-15T15:16:04.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd80061f0 msgr2=0x7fedd80d2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fedd4ff9640 1 --2- 172.21.15.46:0/193467731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fedd80061f0 0x7fedd80d2320 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fedd000b3e0 tx=0x7fedd000b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fede68d5640 1 -- 172.21.15.46:0/193467731 reap_dead start 2024-04-15T15:16:04.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f406c9c0 msgr2=0x7ff2f406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f406c9c0 0x7ff2f406ee80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff30c036000 tx=0x7ff30c03b040 comp rx=0 tx=0).stop 2024-04-15T15:16:04.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7f187f7fe640 1 -- 172.21.15.46:0/572255455 <== osd.7 v2:172.21.15.78:6808/563779653 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1878002dd0 con 0x7f1884076060 2024-04-15T15:16:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3080df2d0 msgr2=0x7ff30803ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 --2- 172.21.15.46:0/1455474953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3080df2d0 0x7ff30803ad10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff310051650 tx=0x7ff3100694c0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3158f6640 1 -- 172.21.15.46:0/1455474953 reap_dead start 2024-04-15T15:16:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 shutdown_connections 2024-04-15T15:16:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 >> 172.21.15.46:0/1455474953 conn(0x7ff30800a400 msgr2=0x7ff3080530c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 shutdown_connections 2024-04-15T15:16:04.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 --> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fea2804fd50 con 0x7fea140760c0 2024-04-15T15:16:04.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 shutdown_connections 2024-04-15T15:16:04.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 >> 172.21.15.46:0/193467731 conn(0x7fedd8058ef0 msgr2=0x7fedd803e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.304+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb1a806c9c0 msgr2=0x7fb1a806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb1a806c9c0 0x7fb1a806ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc007e90 tx=0x7fb1bc02f000 comp rx=0 tx=0).stop 2024-04-15T15:16:04.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1c80ca8b0 msgr2=0x7fb1c80cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 --2- 172.21.15.46:0/3381314023 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb1c80ca8b0 0x7fb1c80cad40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80687d0 tx=0x7fb1c000d810 comp rx=0 tx=0).stop 2024-04-15T15:16:04.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7ff3160f7640 1 -- 172.21.15.46:0/1455474953 wait complete. 2024-04-15T15:16:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 shutdown_connections 2024-04-15T15:16:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f1884076060 msgr2=0x7f18840784a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f1884076060 0x7f18840784a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fedd4ff9640 1 -- 172.21.15.46:0/193467731 wait complete. 2024-04-15T15:16:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1c6d9d640 1 -- 172.21.15.46:0/3381314023 reap_dead start 2024-04-15T15:16:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 shutdown_connections 2024-04-15T15:16:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 >> 172.21.15.46:0/3381314023 conn(0x7fb1c80b3fb0 msgr2=0x7fb1c806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 shutdown_connections 2024-04-15T15:16:04.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fea0ffff640 1 -- 172.21.15.46:0/2589717156 <== osd.0 v2:172.21.15.46:6816/2271822553 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fea2804fd50 con 0x7fea140760c0 2024-04-15T15:16:04.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7fb1ccc86640 1 -- 172.21.15.46:0/3381314023 wait complete. 2024-04-15T15:16:04.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f188406c8f0 msgr2=0x7f188406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f188406c8f0 0x7f188406edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f18880011e0 tx=0x7f1888009040 comp rx=0 tx=0).stop 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1890048d80 msgr2=0x7f189004dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 --2- 172.21.15.46:0/572255455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1890048d80 0x7f189004dce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1898069090 tx=0x7f18980694c0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f18977fe640 1 -- 172.21.15.46:0/572255455 reap_dead start 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 shutdown_connections 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 >> 172.21.15.46:0/572255455 conn(0x7f1890055bc0 msgr2=0x7f18900e8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.305+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 shutdown_connections 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7f187d7fa640 1 -- 172.21.15.46:0/572255455 wait complete. 2024-04-15T15:16:04.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7fea140760c0 msgr2=0x7fea14078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7fea140760c0 0x7fea14078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 --> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdeac002dd0 con 0x7fdea8076040 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea1406c940 msgr2=0x7fea1406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea1406c940 0x7fea1406ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fea1c039000 tx=0x7fea1c034000 comp rx=0 tx=0).stop 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea280b2f50 msgr2=0x7fea2814e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 --2- 172.21.15.46:0/2589717156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea280b2f50 0x7fea2814e0d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea1800d620 tx=0x7fea1800dae0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea27fff640 1 -- 172.21.15.46:0/2589717156 reap_dead start 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 shutdown_connections 2024-04-15T15:16:04.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 >> 172.21.15.46:0/2589717156 conn(0x7fea280b1c50 msgr2=0x7fea28159d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fdeb6ffd640 1 -- 172.21.15.46:0/3170196910 <== osd.2 v2:172.21.15.46:6800/1733872723 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdeac002dd0 con 0x7fdea8076040 2024-04-15T15:16:04.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.306+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 shutdown_connections 2024-04-15T15:16:04.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.307+0000 7fea2cc9e640 1 -- 172.21.15.46:0/2589717156 wait complete. 2024-04-15T15:16:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.307+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fdea8076040 msgr2=0x7fdea8078480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fdea8076040 0x7fdea8078480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdea806c940 msgr2=0x7fdea806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdea806c940 0x7fdea806ee00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000f60 tx=0x7fdeb80079b0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00587e0 msgr2=0x7fdec00e51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 --2- 172.21.15.46:0/3170196910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdec00587e0 0x7fdec00e51f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdebc00c460 tx=0x7fdebc00c920 comp rx=0 tx=0).stop 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec6d9d640 1 -- 172.21.15.46:0/3170196910 reap_dead start 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 shutdown_connections 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 >> 172.21.15.46:0/3170196910 conn(0x7fdec003aa50 msgr2=0x7fdec003d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 shutdown_connections 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.308+0000 7fdec759e640 1 -- 172.21.15.46:0/3170196910 wait complete. 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.309+0000 7fe51e900640 1 -- 172.21.15.46:0/982608088 --> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe5180690f0 con 0x7fe504076070 2024-04-15T15:16:04.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.309+0000 7fe514ff9640 1 -- 172.21.15.46:0/982608088 <== osd.4 v2:172.21.15.78:6800/3745610613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe5180690f0 con 0x7fe504076070 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.316+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7fe504076070 msgr2=0x7fe5040784b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.316+0000 7fe4fe7fc640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7fe504076070 0x7fe5040784b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.316+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe50406c8f0 msgr2=0x7fe50406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.316+0000 7fe4fe7fc640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe50406c8f0 0x7fe50406edb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe508034000 tx=0x7fe508008030 comp rx=0 tx=0).stop 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.316+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe51809a9d0 msgr2=0x7fe5180b5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.316+0000 7fe4fe7fc640 1 --2- 172.21.15.46:0/982608088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe51809a9d0 0x7fe5180b5b70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00e9b0 tx=0x7fe50c00ee70 comp rx=0 tx=0).stop 2024-04-15T15:16:04.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.317+0000 7fe51ce9d640 1 -- 172.21.15.46:0/982608088 reap_dead start 2024-04-15T15:16:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.317+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 shutdown_connections 2024-04-15T15:16:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.317+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 >> 172.21.15.46:0/982608088 conn(0x7fe5180b3ec0 msgr2=0x7fe518070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.317+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 shutdown_connections 2024-04-15T15:16:04.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.318+0000 7fe4fe7fc640 1 -- 172.21.15.46:0/982608088 wait complete. 2024-04-15T15:16:04.383 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:04.383 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-15T15:16:04.384 INFO:teuthology.orchestra.run.smithi046.stdout:60129542155 2024-04-15T15:16:04.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-15T15:16:04.386 INFO:teuthology.orchestra.run.smithi046.stdout:55834574858 2024-04-15T15:16:04.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-15T15:16:04.387 INFO:teuthology.orchestra.run.smithi046.stdout:60129542155 2024-04-15T15:16:04.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-15T15:16:04.388 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:04.388 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:04.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-15T15:16:04.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-15T15:16:04.390 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:04.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-15T15:16:04.391 INFO:teuthology.orchestra.run.smithi046.stdout:60129542154 2024-04-15T15:16:04.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-15T15:16:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.597+0000 7fe3596d9640 1 Processor -- start 2024-04-15T15:16:04.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.597+0000 7fe3596d9640 1 -- start start 2024-04-15T15:16:04.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.598+0000 7fe3596d9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe35416e8c0 0x7fe35416cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.598+0000 7fe3596d9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35416f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.598+0000 7fe3596d9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe35416db00 con 0x7fe3540b3260 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.598+0000 7fe3596d9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe35416f8e0 con 0x7fe35416e8c0 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.598+0000 7fe3596d9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe35416dde0 con 0x7fe35416ef60 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3537fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe3540b3260 0x7fe3540b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47154/0 (socket says 172.21.15.46:47154) 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3537fe640 1 -- 172.21.15.46:0/3799069648 learned_addr learned my addr 172.21.15.46:0/3799069648 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3527fc640 1 --2- 172.21.15.46:0/3799069648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35416f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe352ffd640 1 --2- 172.21.15.46:0/3799069648 >> v2:172.21.15.46:3300/0 conn(0x7fe35416e8c0 0x7fe35416cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.600+0000 7f6fe73e2640 1 Processor -- start 2024-04-15T15:16:04.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3527fc640 1 -- 172.21.15.46:0/3799069648 >> v2:172.21.15.46:3300/0 conn(0x7fe35416e8c0 msgr2=0x7fe35416cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.601+0000 7f6fe73e2640 1 -- start start 2024-04-15T15:16:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3527fc640 1 --2- 172.21.15.46:0/3799069648 >> v2:172.21.15.46:3300/0 conn(0x7fe35416e8c0 0x7fe35416cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3527fc640 1 -- 172.21.15.46:0/3799069648 >> v1:172.21.15.78:6789/0 conn(0x7fe3540b3260 legacy=0x7fe3540b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.599+0000 7fe3527fc640 1 -- 172.21.15.46:0/3799069648 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35416e0c0 con 0x7fe35416ef60 2024-04-15T15:16:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.600+0000 7fe352ffd640 1 --2- 172.21.15.46:0/3799069648 >> v2:172.21.15.46:3300/0 conn(0x7fe35416e8c0 0x7fe35416cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.601+0000 7fe3527fc640 1 --2- 172.21.15.46:0/3799069648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35416f3a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3440097a0 tx=0x7fe34402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b6ee3b8acab252 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.601+0000 7f4a82746640 1 Processor -- start 2024-04-15T15:16:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.601+0000 7f4a82746640 1 -- start start 2024-04-15T15:16:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.601+0000 7f5d69063640 1 Processor -- start 2024-04-15T15:16:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7fe351ffb640 1 -- 172.21.15.46:0/3799069648 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344037070 con 0x7fe35416ef60 2024-04-15T15:16:04.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7fe351ffb640 1 -- 172.21.15.46:0/3799069648 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe344030710 con 0x7fe35416ef60 2024-04-15T15:16:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7fe351ffb640 1 -- 172.21.15.46:0/3799069648 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344030ae0 con 0x7fe35416ef60 2024-04-15T15:16:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.601+0000 7f5d69063640 1 -- start start 2024-04-15T15:16:04.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f89a5073640 1 Processor -- start 2024-04-15T15:16:04.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f5d69063640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5d5c0ead00 0x7f5d5c04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f6fe73e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fe015c410 0x7f6fe00ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f5d69063640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d5c04ffd0 0x7f5d5c037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f5d69063640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d5c053720 con 0x7f5d5c038110 2024-04-15T15:16:04.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7fcc42be8640 1 Processor -- start 2024-04-15T15:16:04.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f6fe73e2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe0072b00 0x7f6fe0072f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f6fe73e2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe00cd710 con 0x7f6fe006eb60 2024-04-15T15:16:04.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f5d69063640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5d5c054330 con 0x7f5d5c04ffd0 2024-04-15T15:16:04.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f6fe73e2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe015b860 con 0x7f6fe015c410 2024-04-15T15:16:04.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f6fe73e2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe00b5cc0 con 0x7f6fe0072b00 2024-04-15T15:16:04.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f5d69063640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f5d5c053a00 con 0x7f5d5c0ead00 2024-04-15T15:16:04.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fcc42be8640 1 -- start start 2024-04-15T15:16:04.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f5d637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d5c04ffd0 0x7f5d5c037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a82746640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c073500 0x7f4a7c073940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a82746640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4a7c0cc860 0x7f4a7c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a82746640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7c0b3150 con 0x7f4a7c15c460 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a82746640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4a7c0ce1c0 con 0x7f4a7c0cc860 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a82746640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c0cd6e0 con 0x7f4a7c073500 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f5d637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5d5c04ffd0 0x7f5d5c037bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42208/0 (socket says 172.21.15.46:42208) 2024-04-15T15:16:04.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f5d637fe640 1 -- 172.21.15.46:0/3685852347 learned_addr learned my addr 172.21.15.46:0/3685852347 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f5d63fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5d5c0ead00 0x7f5d5c04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.602+0000 7f89a5073640 1 -- start start 2024-04-15T15:16:04.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f6fe497d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fe015c410 0x7f6fe00ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fe3596d9640 1 -- 172.21.15.46:0/3799069648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 msgr2=0x7fe35416f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fe3596d9640 1 --2- 172.21.15.46:0/3799069648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35416f3a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe3440097a0 tx=0x7fe34402b790 comp rx=0 tx=0).stop 2024-04-15T15:16:04.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fe3596d9640 1 -- 172.21.15.46:0/3799069648 shutdown_connections 2024-04-15T15:16:04.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fe3596d9640 1 --2- 172.21.15.46:0/3799069648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35416f3a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fe3596d9640 1 --2- 172.21.15.46:0/3799069648 >> v2:172.21.15.46:3300/0 conn(0x7fe35416e8c0 0x7fe35416cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe497d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6fe015c410 0x7f6fe00ce5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42220/0 (socket says 172.21.15.46:42220) 2024-04-15T15:16:04.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f6fe517e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6fe006eb60 0x7f6fe00cded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55644/0 (socket says 172.21.15.46:55644) 2024-04-15T15:16:04.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe517e640 1 -- 172.21.15.46:0/321646920 learned_addr learned my addr 172.21.15.46:0/321646920 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a81744640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4a7c15c460 0x7f4a7c0cf4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55630/0 (socket says 172.21.15.46:55630) 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a81744640 1 -- 172.21.15.46:0/1691091532 learned_addr learned my addr 172.21.15.46:0/1691091532 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a81f45640 1 --2- 172.21.15.46:0/1691091532 >> v2:172.21.15.46:3300/0 conn(0x7f4a7c0cc860 0x7f4a7c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1691091532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c073500 0x7f4a7c073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a73fff640 1 -- 172.21.15.46:0/1691091532 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 710414303 0 0) 0x7f4a7c0b3150 con 0x7f4a7c15c460 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a73fff640 1 -- 172.21.15.46:0/1691091532 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a68003520 con 0x7f4a7c15c460 2024-04-15T15:16:04.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.603+0000 7fe3596d9640 1 -- 172.21.15.46:0/3799069648 >> 172.21.15.46:0/3799069648 conn(0x7fe3540b5680 msgr2=0x7fe3540b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f89a5073640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f89a015e680 0x7f89a006cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f89a5073640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f89a006d2b0 0x7f89a00d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc42be8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc340e3210 0x7fcc34053f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc42be8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc34006220 0x7fcc340581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc42be8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc34056c10 con 0x7fcc34006640 2024-04-15T15:16:04.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc42be8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcc340545f0 con 0x7fcc34006220 2024-04-15T15:16:04.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc42be8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcc34057400 con 0x7fcc340e3210 2024-04-15T15:16:04.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f89a5073640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89a00cef60 con 0x7f89a00d1260 2024-04-15T15:16:04.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f89a5073640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f89a00d1770 con 0x7f89a006d2b0 2024-04-15T15:16:04.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f89a5073640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f89a00d1a50 con 0x7f89a015e680 2024-04-15T15:16:04.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4d92a35640 1 Processor -- start 2024-04-15T15:16:04.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe597f640 1 --2- 172.21.15.46:0/321646920 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe0072b00 0x7f6fe0072f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fe3596d9640 1 -- 172.21.15.46:0/3799069648 shutdown_connections 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fd7fff640 1 -- 172.21.15.46:0/321646920 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3742160634 0 0) 0x7f6fe00cd710 con 0x7f6fe006eb60 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fd7fff640 1 -- 172.21.15.46:0/321646920 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fcc003520 con 0x7f6fe006eb60 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe497d640 1 -- 172.21.15.46:0/321646920 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe0072b00 msgr2=0x7f6fe0072f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe497d640 1 --2- 172.21.15.46:0/321646920 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe0072b00 0x7f6fe0072f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe497d640 1 -- 172.21.15.46:0/321646920 >> v1:172.21.15.46:6789/0 conn(0x7f6fe006eb60 legacy=0x7f6fe00cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f6fe497d640 1 -- 172.21.15.46:0/321646920 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe00cf090 con 0x7f6fe015c410 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f899ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f89a015e680 0x7f89a006cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f899ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f89a015e680 0x7f89a006cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47658/0 (socket says 172.21.15.46:47658) 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f899ffff640 1 -- 172.21.15.46:0/3574695246 learned_addr learned my addr 172.21.15.46:0/3574695246 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc40984640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc340e3210 0x7fcc34053f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc41185640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcc34006640 0x7fcc340e6c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55656/0 (socket says 172.21.15.46:55656) 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc41185640 1 -- 172.21.15.46:0/2376049331 learned_addr learned my addr 172.21.15.46:0/2376049331 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7fcc3bfff640 1 --2- 172.21.15.46:0/2376049331 >> v2:172.21.15.46:3300/0 conn(0x7fcc34006220 0x7fcc340581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f5d637fe640 1 -- 172.21.15.46:0/3685852347 >> v2:172.21.15.78:3300/0 conn(0x7f5d5c0ead00 msgr2=0x7f5d5c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f5d637fe640 1 --2- 172.21.15.46:0/3685852347 >> v2:172.21.15.78:3300/0 conn(0x7f5d5c0ead00 0x7f5d5c04fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f5d637fe640 1 -- 172.21.15.46:0/3685852347 >> v1:172.21.15.78:6789/0 conn(0x7f5d5c038110 legacy=0x7f5d5c038580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f5d637fe640 1 -- 172.21.15.46:0/3685852347 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c053ce0 con 0x7f5d5c04ffd0 2024-04-15T15:16:04.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a80f43640 1 -- 172.21.15.46:0/1691091532 >> v2:172.21.15.46:3300/0 conn(0x7f4a7c0cc860 msgr2=0x7f4a7c0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f4d92a35640 1 -- start start 2024-04-15T15:16:04.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7fe3596d9640 1 -- 172.21.15.46:0/3799069648 wait complete. 2024-04-15T15:16:04.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f899ffff640 1 -- 172.21.15.46:0/3574695246 >> v2:172.21.15.46:3300/0 conn(0x7f89a006d2b0 msgr2=0x7f89a00d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f899ffff640 1 --2- 172.21.15.46:0/3574695246 >> v2:172.21.15.46:3300/0 conn(0x7f89a006d2b0 0x7f89a00d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f899ffff640 1 -- 172.21.15.46:0/3574695246 >> v1:172.21.15.78:6789/0 conn(0x7f89a00d1260 legacy=0x7f89a00d29a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:16:04.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1691091532 >> v2:172.21.15.46:3300/0 conn(0x7f4a7c0cc860 0x7f4a7c0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a80f43640 1 -- 172.21.15.46:0/1691091532 >> v1:172.21.15.46:6789/0 conn(0x7f4a7c15c460 legacy=0x7f4a7c0cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a80f43640 1 -- 172.21.15.46:0/1691091532 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c15c840 con 0x7f4a7c073500 2024-04-15T15:16:04.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.604+0000 7f4a81f45640 1 --2- 172.21.15.46:0/1691091532 >> v2:172.21.15.46:3300/0 conn(0x7f4a7c0cc860 0x7f4a7c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:04.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f899ffff640 1 -- 172.21.15.46:0/3574695246 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a00cf200 con 0x7f89a015e680 2024-04-15T15:16:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1691091532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c073500 0x7f4a7c073940 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7402be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c2f1506dd9ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7fcc3bfff640 1 -- 172.21.15.46:0/2376049331 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc340e3210 msgr2=0x7fcc34053f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7fcc3bfff640 1 --2- 172.21.15.46:0/2376049331 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc340e3210 0x7fcc34053f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7fcc3bfff640 1 -- 172.21.15.46:0/2376049331 >> v1:172.21.15.46:6789/0 conn(0x7fcc34006640 legacy=0x7fcc340e6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7fcc3bfff640 1 -- 172.21.15.46:0/2376049331 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340576e0 con 0x7fcc34006220 2024-04-15T15:16:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7fcc40984640 1 --2- 172.21.15.46:0/2376049331 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc340e3210 0x7fcc34053f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f4a73fff640 1 -- 172.21.15.46:0/1691091532 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74039070 con 0x7f4a7c073500 2024-04-15T15:16:04.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.605+0000 7f4a73fff640 1 -- 172.21.15.46:0/1691091532 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7402cbe0 con 0x7f4a7c073500 2024-04-15T15:16:04.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f6fe497d640 1 --2- 172.21.15.46:0/321646920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe015c410 0x7f6fe00ce5e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0097a0 tx=0x7f6fdc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc9b559ea1edc91 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4d92a35640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4d8bfff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4d8bfff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c06eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42822/0 (socket says 172.21.15.46:42822) 2024-04-15T15:16:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7fcc3bfff640 1 --2- 172.21.15.46:0/2376049331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc340581c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009a00 tx=0x7fcc3002bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c646cc998e8447 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4d90fd2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4d8c15f9f0 0x7f4d8c0cded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55666/0 (socket says 172.21.15.46:55666) 2024-04-15T15:16:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4d92a35640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8c0cd710 con 0x7f4d8c15f9f0 2024-04-15T15:16:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4a82746640 1 -- 172.21.15.46:0/1691091532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c073500 msgr2=0x7f4a7c073940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4a73fff640 1 -- 172.21.15.46:0/1691091532 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a74039070 con 0x7f4a7c073500 2024-04-15T15:16:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.606+0000 7f4a82746640 1 --2- 172.21.15.46:0/1691091532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c073500 0x7f4a7c073940 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7402be30 comp rx=0 tx=0).stop 2024-04-15T15:16:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4d92a35640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8c0cf090 con 0x7f4d8c15c410 2024-04-15T15:16:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4d92a35640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8c06f0a0 con 0x7f4d8c15ef00 2024-04-15T15:16:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4a82746640 1 -- 172.21.15.46:0/1691091532 shutdown_connections 2024-04-15T15:16:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4a82746640 1 --2- 172.21.15.46:0/1691091532 >> v2:172.21.15.46:3300/0 conn(0x7f4a7c0cc860 0x7f4a7c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4a82746640 1 --2- 172.21.15.46:0/1691091532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c073500 0x7f4a7c073940 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4a82746640 1 -- 172.21.15.46:0/1691091532 >> 172.21.15.46:0/1691091532 conn(0x7f4a7c0b1c50 msgr2=0x7f4a7c0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4d8bfff640 1 -- 172.21.15.46:0/3526330613 learned_addr learned my addr 172.21.15.46:0/3526330613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f5d637fe640 1 --2- 172.21.15.46:0/3685852347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c037bd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c007ab0 tx=0x7f5d5002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b505249bf556e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7fcc3b7fe640 1 -- 172.21.15.46:0/2376049331 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002d6e0 con 0x7fcc34006220 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7fcc3b7fe640 1 -- 172.21.15.46:0/2376049331 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc3002d880 con 0x7fcc34006220 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f6fd7fff640 1 -- 172.21.15.46:0/321646920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc031780 con 0x7f6fe015c410 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f6fd7fff640 1 -- 172.21.15.46:0/321646920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc037070 con 0x7f6fe015c410 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f6fd7fff640 1 -- 172.21.15.46:0/321646920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc031b40 con 0x7f6fe015c410 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7fcc3b7fe640 1 -- 172.21.15.46:0/2376049331 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002dc90 con 0x7fcc34006220 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f899ffff640 1 --2- 172.21.15.46:0/3574695246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a006cd70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f89a006e7f0 tx=0x7f899002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c255059fae77966e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7fe3596d9640 1 Processor -- start 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7fe3596d9640 1 -- start start 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f899f7fe640 1 -- 172.21.15.46:0/3574695246 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899002e070 con 0x7f89a015e680 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f899f7fe640 1 -- 172.21.15.46:0/3574695246 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8990037070 con 0x7f89a015e680 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f899f7fe640 1 -- 172.21.15.46:0/3574695246 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990031a00 con 0x7f89a015e680 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f5d62ffd640 1 -- 172.21.15.46:0/3685852347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50034070 con 0x7f5d5c04ffd0 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f5d62ffd640 1 -- 172.21.15.46:0/3685852347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d50030870 con 0x7f5d5c04ffd0 2024-04-15T15:16:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4a82746640 1 -- 172.21.15.46:0/1691091532 shutdown_connections 2024-04-15T15:16:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4a82746640 1 -- 172.21.15.46:0/1691091532 wait complete. 2024-04-15T15:16:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fcc42be8640 1 -- 172.21.15.46:0/2376049331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 msgr2=0x7fcc340581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fcc42be8640 1 --2- 172.21.15.46:0/2376049331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc340581c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009a00 tx=0x7fcc3002bb30 comp rx=0 tx=0).stop 2024-04-15T15:16:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857837705 0 0) 0x7f4d8c0cd710 con 0x7f4d8c15f9f0 2024-04-15T15:16:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.607+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d78003520 con 0x7f4d8c15f9f0 2024-04-15T15:16:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3596d9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3540b3260 0x7fe3540cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fd25f59e640 1 Processor -- start 2024-04-15T15:16:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3596d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 0x7fe354094b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3596d9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35408c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3596d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe35416d4c0 con 0x7fe35416e8c0 2024-04-15T15:16:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3596d9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe35416eca0 con 0x7fe3540b3260 2024-04-15T15:16:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3596d9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe3540b3de0 con 0x7fe35416ef60 2024-04-15T15:16:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fd25f59e640 1 -- start start 2024-04-15T15:16:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 0x7fe354094b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 0x7fe354094b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42252/0 (socket says 172.21.15.46:42252) 2024-04-15T15:16:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3527fc640 1 -- 172.21.15.46:0/754083747 learned_addr learned my addr 172.21.15.46:0/754083747 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe352ffd640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3540b3260 0x7fe3540cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fcc42be8640 1 -- 172.21.15.46:0/2376049331 shutdown_connections 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fcc42be8640 1 --2- 172.21.15.46:0/2376049331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc340581c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fcc42be8640 1 --2- 172.21.15.46:0/2376049331 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc340e3210 0x7fcc34053f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fcc42be8640 1 -- 172.21.15.46:0/2376049331 >> 172.21.15.46:0/2376049331 conn(0x7fcc34058f20 msgr2=0x7fcc34059330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f5d69063640 1 -- 172.21.15.46:0/3685852347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c04ffd0 msgr2=0x7f5d5c037bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f5d69063640 1 --2- 172.21.15.46:0/3685852347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c037bd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c007ab0 tx=0x7f5d5002b9b0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f5d62ffd640 1 -- 172.21.15.46:0/3685852347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50030c40 con 0x7f5d5c04ffd0 2024-04-15T15:16:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1727854708 0 0) 0x7f4d8c0cf090 con 0x7f4d8c15c410 2024-04-15T15:16:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d8c0cd710 con 0x7f4d8c15c410 2024-04-15T15:16:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 345289979 0 0) 0x7f4d78003520 con 0x7f4d8c15f9f0 2024-04-15T15:16:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f4d8bfff640 1 -- 172.21.15.46:0/3526330613 >> v1:172.21.15.78:6789/0 conn(0x7f4d8c15c410 legacy=0x7f4d8c15f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f89a5073640 1 -- 172.21.15.46:0/3574695246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 msgr2=0x7f89a006cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7f89a5073640 1 --2- 172.21.15.46:0/3574695246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a006cd70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f89a006e7f0 tx=0x7f899002b880 comp rx=0 tx=0).stop 2024-04-15T15:16:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f4d8bfff640 1 -- 172.21.15.46:0/3526330613 >> v1:172.21.15.46:6789/0 conn(0x7f4d8c15f9f0 legacy=0x7f4d8c0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f89a5073640 1 -- 172.21.15.46:0/3574695246 shutdown_connections 2024-04-15T15:16:04.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.608+0000 7fe3537fe640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35408c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f89a5073640 1 --2- 172.21.15.46:0/3574695246 >> v2:172.21.15.46:3300/0 conn(0x7f89a006d2b0 0x7f89a00d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f5d69063640 1 -- 172.21.15.46:0/3685852347 shutdown_connections 2024-04-15T15:16:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f5d69063640 1 --2- 172.21.15.46:0/3685852347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c037bd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f6fe73e2640 1 -- 172.21.15.46:0/321646920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe015c410 msgr2=0x7f6fe00ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/321646920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe015c410 0x7f6fe00ce5e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0097a0 tx=0x7f6fdc02b820 comp rx=0 tx=0).stop 2024-04-15T15:16:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f5d69063640 1 --2- 172.21.15.46:0/3685852347 >> v2:172.21.15.78:3300/0 conn(0x7f5d5c0ead00 0x7f5d5c04fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f5d69063640 1 -- 172.21.15.46:0/3685852347 >> 172.21.15.46:0/3685852347 conn(0x7f5d5c009fb0 msgr2=0x7f5d5c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f89a5073640 1 --2- 172.21.15.46:0/3574695246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a006cd70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f89a5073640 1 -- 172.21.15.46:0/3574695246 >> 172.21.15.46:0/3574695246 conn(0x7f89a009d740 msgr2=0x7f89a008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fe3527fc640 1 -- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 msgr2=0x7fe35408c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fe3527fc640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35408c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fe3527fc640 1 -- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3540b3260 msgr2=0x7fe3540cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fe3527fc640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3540b3260 0x7fe3540cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fe3527fc640 1 -- 172.21.15.46:0/754083747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3540cfe10 con 0x7fe35416e8c0 2024-04-15T15:16:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fcc42be8640 1 -- 172.21.15.46:0/2376049331 shutdown_connections 2024-04-15T15:16:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f5d69063640 1 -- 172.21.15.46:0/3685852347 shutdown_connections 2024-04-15T15:16:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f6fe73e2640 1 -- 172.21.15.46:0/321646920 shutdown_connections 2024-04-15T15:16:04.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/321646920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe015c410 0x7f6fe00ce5e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/321646920 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe0072b00 0x7f6fe0072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f6fe73e2640 1 -- 172.21.15.46:0/321646920 >> 172.21.15.46:0/321646920 conn(0x7f6fe00b2c60 msgr2=0x7f6fe00b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f5d69063640 1 -- 172.21.15.46:0/3685852347 wait complete. 2024-04-15T15:16:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f4a82746640 1 Processor -- start 2024-04-15T15:16:04.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f89a5073640 1 -- 172.21.15.46:0/3574695246 shutdown_connections 2024-04-15T15:16:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fe3527fc640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 0x7fe354094b30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe34402bda0 tx=0x7fe3440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fd25f59e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2580dfd30 0x7fd258053ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fd25f59e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258038d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f89a5073640 1 -- 172.21.15.46:0/3574695246 wait complete. 2024-04-15T15:16:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25f59e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd258054530 con 0x7fd25803b560 2024-04-15T15:16:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25f59e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd2580547d0 con 0x7fd2580dfd30 2024-04-15T15:16:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25f59e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd258054ab0 con 0x7fd258050fb0 2024-04-15T15:16:04.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7fcc42be8640 1 -- 172.21.15.46:0/2376049331 wait complete. 2024-04-15T15:16:04.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f4d8bfff640 1 -- 172.21.15.46:0/3526330613 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c0cf370 con 0x7f4d8c15ef00 2024-04-15T15:16:04.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25dd9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd25803b560 0x7fd2580508a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:47198/0 (socket says 172.21.15.46:47198) 2024-04-15T15:16:04.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25dd9b640 1 -- 172.21.15.46:0/2256973820 learned_addr learned my addr 172.21.15.46:0/2256973820 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:04.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25ed9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7f6fe73e2640 1 -- 172.21.15.46:0/321646920 shutdown_connections 2024-04-15T15:16:04.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7f4d8bfff640 1 --2- 172.21.15.46:0/3526330613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c06eb60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009820 tx=0x7f4d7c02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90da3380076fc320 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.609+0000 7f4a82746640 1 -- start start 2024-04-15T15:16:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7f4a82746640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 0x7f4a7c15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7f4a82746640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c0cc860 0x7f4a7c0b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a82746640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a7c15c460 0x7f4a7c13e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a82746640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c0ce9d0 con 0x7f4a7c15c460 2024-04-15T15:16:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a82746640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c0ce6f0 con 0x7f4a7c073500 2024-04-15T15:16:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a82746640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c0cecb0 con 0x7f4a7c0cc860 2024-04-15T15:16:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25ed9d640 1 -- 172.21.15.46:0/2256973820 >> v2:172.21.15.78:3300/0 conn(0x7fd2580dfd30 msgr2=0x7fd258053ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:16:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25ed9d640 1 --2- 172.21.15.46:0/2256973820 >> v2:172.21.15.78:3300/0 conn(0x7fd2580dfd30 0x7fd258053ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25ed9d640 1 -- 172.21.15.46:0/2256973820 >> v1:172.21.15.78:6789/0 conn(0x7fd25803b560 legacy=0x7fd2580508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fd25ed9d640 1 -- 172.21.15.46:0/2256973820 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd258054d90 con 0x7fd258050fb0 2024-04-15T15:16:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 0x7f4a7c15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 0x7f4a7c15ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47674/0 (socket says 172.21.15.46:47674) 2024-04-15T15:16:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344041020 con 0x7fe35416e8c0 2024-04-15T15:16:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe344032cd0 con 0x7fe35416e8c0 2024-04-15T15:16:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe344037040 con 0x7fe35416e8c0 2024-04-15T15:16:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.610+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3540d00f0 con 0x7fe35416e8c0 2024-04-15T15:16:04.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7fcc42be8640 1 Processor -- start 2024-04-15T15:16:04.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c037070 con 0x7f4d8c15ef00 2024-04-15T15:16:04.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d7c030510 con 0x7f4d8c15ef00 2024-04-15T15:16:04.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4d8b7fe640 1 -- 172.21.15.46:0/3526330613 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d840019f0 con 0x7f4d8c15ef00 2024-04-15T15:16:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 -- 172.21.15.46:0/1259485852 learned_addr learned my addr 172.21.15.46:0/1259485852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c0cc860 0x7f4a7c0b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3540d06b0 con 0x7fe35416e8c0 2024-04-15T15:16:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 -- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c0cc860 msgr2=0x7f4a7c0b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81f45640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a7c15c460 0x7f4a7c13e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c0cc860 0x7f4a7c0b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f89a5073640 1 Processor -- start 2024-04-15T15:16:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 -- 172.21.15.46:0/1259485852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a7c15c460 msgr2=0x7f4a7c13e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a7c15c460 0x7f4a7c13e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f89a5073640 1 -- start start 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a81744640 1 -- 172.21.15.46:0/1259485852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c1496c0 con 0x7f4a7c073500 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c0cc860 0x7f4a7c0b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4a81f45640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a7c15c460 0x7f4a7c13e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4a81744640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 0x7f4a7c15ca30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c3c0 tx=0x7f4a7800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f5d69063640 1 Processor -- start 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f5d69063640 1 -- start start 2024-04-15T15:16:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7fcc42be8640 1 -- start start 2024-04-15T15:16:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4d92a35640 1 -- 172.21.15.46:0/3526330613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 msgr2=0x7f4d8c06eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3526330613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c06eb60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009820 tx=0x7f4d7c02b7c0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a780075a0 con 0x7f4a7c073500 2024-04-15T15:16:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a780047e0 con 0x7f4a7c073500 2024-04-15T15:16:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78004c10 con 0x7f4a7c073500 2024-04-15T15:16:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7c1499a0 con 0x7f4a7c073500 2024-04-15T15:16:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4d92a35640 1 -- 172.21.15.46:0/3526330613 shutdown_connections 2024-04-15T15:16:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3526330613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c06eb60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7f4d92a35640 1 -- 172.21.15.46:0/3526330613 >> 172.21.15.46:0/3526330613 conn(0x7f4d8c0b2c60 msgr2=0x7f4d8c0b3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7fcc42be8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc3403a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7fcc42be8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc34006640 0x7fcc3417df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7fcc42be8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 0x7fcc3417a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7fcc42be8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34058700 con 0x7fcc34006220 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7fcc42be8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34057ca0 con 0x7fcc340e3210 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.612+0000 7fcc42be8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcc34058880 con 0x7fcc34006640 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc3bfff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc34006640 0x7fcc3417df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 0x7fcc3417a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7c0b5020 con 0x7f4a7c073500 2024-04-15T15:16:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 0x7fcc3417a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:47690/0 (socket says 172.21.15.46:47690) 2024-04-15T15:16:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 -- 172.21.15.46:0/4225106054 learned_addr learned my addr 172.21.15.46:0/4225106054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d69063640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 0x7f5d5c0dd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d69063640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c048ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d69063640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c0ead00 0x7f5d5c0493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d69063640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c0547c0 con 0x7f5d5c0ead00 2024-04-15T15:16:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d69063640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c038c90 con 0x7f5d5c04ffd0 2024-04-15T15:16:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d69063640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d5c054940 con 0x7f5d5c038110 2024-04-15T15:16:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f89a5073640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 0x7f89a00c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f89a5073640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a00d1260 0x7f89a00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.611+0000 7f6fe73e2640 1 -- 172.21.15.46:0/321646920 wait complete. 2024-04-15T15:16:04.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f6fe73e2640 1 Processor -- start 2024-04-15T15:16:04.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe344044020 con 0x7fe35416e8c0 2024-04-15T15:16:04.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d63fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 0x7f5d5c0dd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d63fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 0x7f5d5c0dd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42856/0 (socket says 172.21.15.46:42856) 2024-04-15T15:16:04.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d63fff640 1 -- 172.21.15.46:0/4046234100 learned_addr learned my addr 172.21.15.46:0/4046234100 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f5d68862640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c0ead00 0x7f5d5c0493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f4d92a35640 1 -- 172.21.15.46:0/3526330613 shutdown_connections 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f4d92a35640 1 -- 172.21.15.46:0/3526330613 wait complete. 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fd25ed9d640 1 --2- 172.21.15.46:0/2256973820 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258038d70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd260051700 tx=0x7fd2600ab590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38c910264d631c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc40984640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc3403a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 -- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc34006640 msgr2=0x7fcc3417df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc34006640 0x7fcc3417df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 -- 172.21.15.46:0/4225106054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 msgr2=0x7fcc3403a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc3403a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7fcc41185640 1 -- 172.21.15.46:0/4225106054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc3417e440 con 0x7fcc340e3210 2024-04-15T15:16:04.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f89a5073640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a00cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fd25d59a640 1 -- 172.21.15.46:0/2256973820 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600b7070 con 0x7fd258050fb0 2024-04-15T15:16:04.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f89a5073640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f89a00d2170 con 0x7f89a00d1260 2024-04-15T15:16:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fd25d59a640 1 -- 172.21.15.46:0/2256973820 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2600b0810 con 0x7fd258050fb0 2024-04-15T15:16:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fd25d59a640 1 -- 172.21.15.46:0/2256973820 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600b7070 con 0x7fd258050fb0 2024-04-15T15:16:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f89a5073640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f89a015ea60 con 0x7f89a015e680 2024-04-15T15:16:04.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f89a5073640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f89a00d22f0 con 0x7f89a006d2b0 2024-04-15T15:16:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fcc3bfff640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc34006640 0x7fcc3417df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fcc41185640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 0x7fcc3417a8e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c3c0 tx=0x7fcc2800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fcc40984640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc3403a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 0x7f89a00c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 0x7f89a00c81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42858/0 (socket says 172.21.15.46:42858) 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 -- 172.21.15.46:0/76492716 learned_addr learned my addr 172.21.15.46:0/76492716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fe33b7fe640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe34006c970 0x7fe34006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f89a4872640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a00cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 -- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 msgr2=0x7f89a00cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a00cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 -- 172.21.15.46:0/76492716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a00d1260 msgr2=0x7f89a00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a00d1260 0x7f89a00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f899ffff640 1 -- 172.21.15.46:0/76492716 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a00b77b0 con 0x7f89a006d2b0 2024-04-15T15:16:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc28017070 con 0x7fcc340e3210 2024-04-15T15:16:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fcc42be8640 1 -- 172.21.15.46:0/4225106054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc34038f70 con 0x7fcc340e3210 2024-04-15T15:16:04.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fcc42be8640 1 -- 172.21.15.46:0/4225106054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc34039530 con 0x7fcc340e3210 2024-04-15T15:16:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7fe352ffd640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe34006c970 0x7fe34006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe344078960 con 0x7fe35416e8c0 2024-04-15T15:16:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3540b48b0 con 0x7fe35416e8c0 2024-04-15T15:16:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a7c04fd50 con 0x7f4a7c073500 2024-04-15T15:16:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f5d68862640 1 -- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 msgr2=0x7f5d5c0dd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f5d68862640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 0x7f5d5c0dd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f5d68862640 1 -- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d5c04ffd0 msgr2=0x7f5d5c048ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.614+0000 7f5d637fe640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c048ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f5d68862640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c048ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f899ffff640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 0x7f89a00c81e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89900090b0 tx=0x7f89900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990032880 con 0x7f89a006d2b0 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fd25f59e640 1 -- 172.21.15.46:0/2256973820 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 msgr2=0x7fd258038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fd25f59e640 1 --2- 172.21.15.46:0/2256973820 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258038d70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd260051700 tx=0x7fd2600ab590 comp rx=0 tx=0).stop 2024-04-15T15:16:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fd25f59e640 1 -- 172.21.15.46:0/2256973820 shutdown_connections 2024-04-15T15:16:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fd25f59e640 1 --2- 172.21.15.46:0/2256973820 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258038d70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fd25f59e640 1 --2- 172.21.15.46:0/2256973820 >> v2:172.21.15.78:3300/0 conn(0x7fd2580dfd30 0x7fd258053ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fd25f59e640 1 -- 172.21.15.46:0/2256973820 >> 172.21.15.46:0/2256973820 conn(0x7fd258055bd0 msgr2=0x7fd258055fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f5d68862640 1 -- 172.21.15.46:0/4046234100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c172530 con 0x7f5d5c0ead00 2024-04-15T15:16:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8990032a20 con 0x7f89a006d2b0 2024-04-15T15:16:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990032df0 con 0x7f89a006d2b0 2024-04-15T15:16:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f5d63fff640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 0x7f5d5c0dd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f5d637fe640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c048ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.613+0000 7f6fe73e2640 1 -- start start 2024-04-15T15:16:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc28005da0 con 0x7fcc340e3210 2024-04-15T15:16:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc280072d0 con 0x7fcc340e3210 2024-04-15T15:16:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f4d92a35640 1 Processor -- start 2024-04-15T15:16:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.615+0000 7f4d92a35640 1 -- start start 2024-04-15T15:16:04.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a00c8720 con 0x7f89a006d2b0 2024-04-15T15:16:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe73e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fe006eb60 0x7f6fe02057a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe73e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe0072b00 0x7f6fe009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe73e2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 0x7f6fe006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a00b71f0 con 0x7f89a006d2b0 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe73e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00ceb20 con 0x7f6fe0072b00 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe73e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00cda40 con 0x7f6fe006eb60 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe73e2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe015c850 con 0x7f6fe009bd80 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 0x7f6fe006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe517e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fe006eb60 0x7f6fe02057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 0x7f6fe006bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42866/0 (socket says 172.21.15.46:42866) 2024-04-15T15:16:04.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 -- 172.21.15.46:0/3906685459 learned_addr learned my addr 172.21.15.46:0/3906685459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7fcc42be8640 1 -- 172.21.15.46:0/4225106054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c005290 con 0x7fcc340e3210 2024-04-15T15:16:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f5d68862640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c0ead00 0x7f5d5c0493e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800c3c0 tx=0x7f5d5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 -- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fe006eb60 msgr2=0x7f6fe02057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fe006eb60 0x7f6fe02057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 -- 172.21.15.46:0/3906685459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe0072b00 msgr2=0x7f6fe009a9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:16:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe0072b00 0x7f6fe009a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f6fe597f640 1 -- 172.21.15.46:0/3906685459 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe009b170 con 0x7f6fe009bd80 2024-04-15T15:16:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f4d92a35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d8c15c410 0x7f4d8c0ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.616+0000 7f4d92a35640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c1528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d92a35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c15f9f0 0x7f4d8c159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f6fe597f640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 0x7f6fe006bc50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8007e90 tx=0x7f6fd800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d92a35640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c0b5cc0 con 0x7f4d8c15f9f0 2024-04-15T15:16:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d92a35640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8c15b860 con 0x7f4d8c15c410 2024-04-15T15:16:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d92a35640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8c0cda40 con 0x7f4d8c15ef00 2024-04-15T15:16:04.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7fe352ffd640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe34006c970 0x7fe34006ee30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe348006fd0 tx=0x7fe348008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7fd25f59e640 1 -- 172.21.15.46:0/2256973820 shutdown_connections 2024-04-15T15:16:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7fd25f59e640 1 -- 172.21.15.46:0/2256973820 wait complete. 2024-04-15T15:16:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d83fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c1528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d83fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c1528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:42882/0 (socket says 172.21.15.46:42882) 2024-04-15T15:16:04.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f4d83fff640 1 -- 172.21.15.46:0/3581990255 learned_addr learned my addr 172.21.15.46:0/3581990255 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8014420 con 0x7f6fe009bd80 2024-04-15T15:16:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd8019070 con 0x7f6fe009bd80 2024-04-15T15:16:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8014820 con 0x7f6fe009bd80 2024-04-15T15:16:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe009b450 con 0x7f6fe009bd80 2024-04-15T15:16:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.617+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe014bf20 con 0x7f6fe009bd80 2024-04-15T15:16:04.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcc2801a070 con 0x7fcc340e3210 2024-04-15T15:16:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8984005290 con 0x7f89a006d2b0 2024-04-15T15:16:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d8bfff640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d8c15c410 0x7f4d8c0ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d83fff640 1 -- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d8c15c410 msgr2=0x7f4d8c0ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d83fff640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d8c15c410 0x7f4d8c0ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d83fff640 1 -- 172.21.15.46:0/3581990255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c15f9f0 msgr2=0x7f4d8c159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d83fff640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c15f9f0 0x7f4d8c159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d83fff640 1 -- 172.21.15.46:0/3581990255 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8c0adf40 con 0x7f4d8c15ef00 2024-04-15T15:16:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7fcc397fa640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcc2006c940 0x7fcc2006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4d83fff640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c1528f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400cb20 tx=0x7f4d74007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74018070 con 0x7f4d8c15ef00 2024-04-15T15:16:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d74007050 con 0x7f4d8c15ef00 2024-04-15T15:16:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d8c15a260 con 0x7f4d8c15ef00 2024-04-15T15:16:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d74007440 con 0x7f4d8c15ef00 2024-04-15T15:16:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7fcc40984640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcc2006c940 0x7fcc2006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7fd25f59e640 1 Processor -- start 2024-04-15T15:16:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d8c0ae1e0 con 0x7f4d8c15ef00 2024-04-15T15:16:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7fd25f59e640 1 -- start start 2024-04-15T15:16:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcc28056b50 con 0x7fcc340e3210 2024-04-15T15:16:04.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe004fd50 con 0x7f6fe009bd80 2024-04-15T15:16:04.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.619+0000 7fcc40984640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcc2006c940 0x7fcc2006ee00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c06aee0 tx=0x7fcc3c06f040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6fd80149c0 con 0x7f6fe009bd80 2024-04-15T15:16:04.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a78011070 con 0x7f4a7c073500 2024-04-15T15:16:04.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4a71ffb640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a6006c9f0 0x7f4a6006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.618+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4a7808b990 con 0x7f4a7c073500 2024-04-15T15:16:04.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a6006c9f0 0x7f4a6006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7f6fd5ffb640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6fc406c9c0 0x7f6fc406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d8c04fd50 con 0x7f4d8c15ef00 2024-04-15T15:16:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7fd25f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 0x7fd25804b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7fd25f59e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258039240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7f4a80f43640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a6006c9f0 0x7f4a6006eeb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4a74036000 tx=0x7f4a74030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6fd808c700 con 0x7f6fe009bd80 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7fd25f59e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2580dfd30 0x7fd258039780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7fd25f59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd258055870 con 0x7fd25803b560 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7fd25f59e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd258055350 con 0x7fd2580dfd30 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.620+0000 7fd25f59e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd25804aa60 con 0x7fd258050fb0 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 0x7fd25804b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25dd9b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258039240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 0x7fd25804b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42312/0 (socket says 172.21.15.46:42312) 2024-04-15T15:16:04.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 -- 172.21.15.46:0/4156764460 learned_addr learned my addr 172.21.15.46:0/4156764460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f899004c060 con 0x7f89a006d2b0 2024-04-15T15:16:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f6fe517e640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6fc406c9c0 0x7f6fc406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25ed9d640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2580dfd30 0x7fd258039780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d7401b070 con 0x7f4d8c15ef00 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 -- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 msgr2=0x7fd258039240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258039240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 -- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2580dfd30 msgr2=0x7fd258039780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2580dfd30 0x7fd258039780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 -- 172.21.15.46:0/4156764460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2580d7830 con 0x7fd25803b560 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25ed9d640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2580dfd30 0x7fd258039780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7fd25e59c640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 0x7fd25804b600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd25000b3e0 tx=0x7fd25000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f6fe517e640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6fc406c9c0 0x7f6fc406ee80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0007fd0 tx=0x7f6fd0007d60 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f899d7fa640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f898006c9c0 0x7f898006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f4d897fa640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.621+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f89900adbd0 con 0x7f89a006d2b0 2024-04-15T15:16:04.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.622+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4d7408b6a0 con 0x7f4d8c15ef00 2024-04-15T15:16:04.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.622+0000 7f8997fff640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f898006c9c0 0x7f898006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.622+0000 7f4d8bfff640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.622+0000 7f8997fff640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f898006c9c0 0x7f898006ee80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f89a015cb10 tx=0x7f8988006bf0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.622+0000 7f4d8bfff640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c034000 tx=0x7f4d7c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc3417e440 con 0x7fcc340e3210 2024-04-15T15:16:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d580052d0 con 0x7f5d5c0ead00 2024-04-15T15:16:04.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250015020 con 0x7fd25803b560 2024-04-15T15:16:04.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2580d7b10 con 0x7fd25803b560 2024-04-15T15:16:04.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd250011830 con 0x7fd25803b560 2024-04-15T15:16:04.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5c172810 con 0x7f5d5c0ead00 2024-04-15T15:16:04.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5c171800 con 0x7f5d5c0ead00 2024-04-15T15:16:04.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3540b48b0 con 0x7fe35416e8c0 2024-04-15T15:16:04.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.623+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2580d80d0 con 0x7fd25803b560 2024-04-15T15:16:04.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.624+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250011c20 con 0x7fd25803b560 2024-04-15T15:16:04.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.624+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d58005470 con 0x7f5d5c0ead00 2024-04-15T15:16:04.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.624+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58007260 con 0x7f5d5c0ead00 2024-04-15T15:16:04.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.625+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd258005710 con 0x7fd25803b560 2024-04-15T15:16:04.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.625+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d44005290 con 0x7f5d5c0ead00 2024-04-15T15:16:04.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.625+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89900ade20 con 0x7f89a006d2b0 2024-04-15T15:16:04.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.626+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd250018020 con 0x7fd25803b560 2024-04-15T15:16:04.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.626+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d58007400 con 0x7f5d5c0ead00 2024-04-15T15:16:04.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.626+0000 7fd24effd640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd24006c8f0 0x7fd24006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.626+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd25008bdf0 con 0x7fd25803b560 2024-04-15T15:16:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.626+0000 7fd25dd9b640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd24006c8f0 0x7fd24006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.627+0000 7f5d60ff9640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.627+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5d5808b150 con 0x7f5d5c0ead00 2024-04-15T15:16:04.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.627+0000 7f5d63fff640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.627+0000 7fd25dd9b640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd24006c8f0 0x7fd24006edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2580d7480 tx=0x7fd254007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.627+0000 7f5d63fff640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5d640654e0 tx=0x7f5d640747a0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.627+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a78059af0 con 0x7f4a7c073500 2024-04-15T15:16:04.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.630+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fd805a7d0 con 0x7f6fe009bd80 2024-04-15T15:16:04.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.635+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d740590a0 con 0x7f4d8c15ef00 2024-04-15T15:16:04.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.635+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd25001f020 con 0x7fd25803b560 2024-04-15T15:16:04.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:04.645+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d580592d0 con 0x7f5d5c0ead00 2024-04-15T15:16:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.040+0000 7fcc42be8640 1 -- 172.21.15.46:0/4225106054 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fcc1c005850 con 0x7fcc340e3210 2024-04-15T15:16:05.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.041+0000 7fcc397fa640 1 -- 172.21.15.46:0/4225106054 <== mon.1 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc3417e440 con 0x7fcc340e3210 2024-04-15T15:16:05.043 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.044+0000 7fcc26ffd640 1 -- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcc2006c940 msgr2=0x7fcc2006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.044+0000 7fcc26ffd640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcc2006c940 0x7fcc2006ee00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c06aee0 tx=0x7fcc3c06f040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 -- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 msgr2=0x7fcc3417a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 0x7fcc3417a8e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c3c0 tx=0x7fcc2800c880 comp rx=0 tx=0).stop 2024-04-15T15:16:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 -- 172.21.15.46:0/4225106054 shutdown_connections 2024-04-15T15:16:05.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcc2006c940 0x7fcc2006ee00 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c06aee0 tx=0x7fcc3c06f040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc340e3210 0x7fcc3417a8e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcc34006640 0x7fcc3417df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 --2- 172.21.15.46:0/4225106054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc34006220 0x7fcc3403a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.045+0000 7fcc26ffd640 1 -- 172.21.15.46:0/4225106054 >> 172.21.15.46:0/4225106054 conn(0x7fcc34058f20 msgr2=0x7fcc3416dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.046+0000 7fcc26ffd640 1 -- 172.21.15.46:0/4225106054 shutdown_connections 2024-04-15T15:16:05.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.046+0000 7fcc26ffd640 1 -- 172.21.15.46:0/4225106054 wait complete. 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.057+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4a7c0cc300 con 0x7f4a7c073500 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.058+0000 7f4a71ffb640 1 -- 172.21.15.46:0/1259485852 <== mon.1 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4a780593f0 con 0x7f4a7c073500 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi046.stdout:55834574858 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.060+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6fe00b3970 con 0x7f6fe009bd80 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.060+0000 7f6fd5ffb640 1 -- 172.21.15.46:0/3906685459 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6fd805a0d0 con 0x7f6fe009bd80 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.061+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a6006c9f0 msgr2=0x7f4a6006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.061+0000 7f4a82746640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a6006c9f0 0x7f4a6006eeb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4a74036000 tx=0x7f4a74030000 comp rx=0 tx=0).stop 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 msgr2=0x7f4a7c15ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 0x7f4a7c15ca30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c3c0 tx=0x7f4a7800c880 comp rx=0 tx=0).stop 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 shutdown_connections 2024-04-15T15:16:05.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-04-15T15:16:05.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8984005850 con 0x7f89a006d2b0 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd2580491d0 con 0x7fd25803b560 2024-04-15T15:16:05.066 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a6006c9f0 0x7f4a6006eeb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a7c15c460 0x7f4a7c13e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a7c0cc860 0x7f4a7c0b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 --2- 172.21.15.46:0/1259485852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a7c073500 0x7f4a7c15ca30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f899d7fa640 1 -- 172.21.15.46:0/76492716 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f89a00b77b0 con 0x7f89a006d2b0 2024-04-15T15:16:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 >> 172.21.15.46:0/1259485852 conn(0x7f4a7c0b1c50 msgr2=0x7f4a7c074da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.062+0000 7fd24effd640 1 -- 172.21.15.46:0/4156764460 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd250059e60 con 0x7fd25803b560 2024-04-15T15:16:05.068 INFO:teuthology.orchestra.run.smithi046.stdout:60129542154 2024-04-15T15:16:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.063+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 shutdown_connections 2024-04-15T15:16:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.063+0000 7f4a82746640 1 -- 172.21.15.46:0/1259485852 wait complete. 2024-04-15T15:16:05.069 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6fc406c9c0 msgr2=0x7f6fc406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6fc406c9c0 0x7f6fc406ee80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0007fd0 tx=0x7f6fd0007d60 comp rx=0 tx=0).stop 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 msgr2=0x7f6fe006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 0x7f6fe006bc50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8007e90 tx=0x7f6fd800e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 shutdown_connections 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6fc406c9c0 0x7f6fc406ee80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6fe009bd80 0x7f6fe006bc50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fe0072b00 0x7f6fe009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 --2- 172.21.15.46:0/3906685459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6fe006eb60 0x7f6fe02057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 >> 172.21.15.46:0/3906685459 conn(0x7f6fe00b2c60 msgr2=0x7f6fe00cded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 shutdown_connections 2024-04-15T15:16:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.064+0000 7f6fe73e2640 1 -- 172.21.15.46:0/3906685459 wait complete. 2024-04-15T15:16:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.065+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd24006c8f0 msgr2=0x7fd24006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.065+0000 7fd25f59e640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd24006c8f0 0x7fd24006edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd2580d7480 tx=0x7fd254007e80 comp rx=0 tx=0).stop 2024-04-15T15:16:05.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.065+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 msgr2=0x7fd25804b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.065+0000 7fd25f59e640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 0x7fd25804b600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd25000b3e0 tx=0x7fd25000b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 shutdown_connections 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd24006c8f0 0x7fd24006edb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2580dfd30 0x7fd258039780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd258050fb0 0x7fd258039240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 --2- 172.21.15.46:0/4156764460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25803b560 0x7fd25804b600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 >> 172.21.15.46:0/4156764460 conn(0x7fd258055bd0 msgr2=0x7fd2580db320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f898006c9c0 msgr2=0x7f898006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f898006c9c0 0x7f898006ee80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f89a015cb10 tx=0x7f8988006bf0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 shutdown_connections 2024-04-15T15:16:05.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 msgr2=0x7f89a00c81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 0x7f89a00c81e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89900090b0 tx=0x7f89900046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 shutdown_connections 2024-04-15T15:16:05.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7fd25f59e640 1 -- 172.21.15.46:0/4156764460 wait complete. 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f898006c9c0 0x7f898006ee80 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f89a015cb10 tx=0x7f8988006bf0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89a015e680 0x7f89a00cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a00d1260 0x7f89a00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 --2- 172.21.15.46:0/76492716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f89a006d2b0 0x7f89a00c81e0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f89900090b0 tx=0x7f89900046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 >> 172.21.15.46:0/76492716 conn(0x7f89a009d740 msgr2=0x7f89a00b3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 shutdown_connections 2024-04-15T15:16:05.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.066+0000 7f89a5073640 1 -- 172.21.15.46:0/76492716 wait complete. 2024-04-15T15:16:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.067+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4d8c0b4690 con 0x7f4d8c15ef00 2024-04-15T15:16:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.068+0000 7f4d897fa640 1 -- 172.21.15.46:0/3581990255 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4d7405d3a0 con 0x7f4d8c15ef00 2024-04-15T15:16:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.067+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe3540689e0 con 0x7fe35416e8c0 2024-04-15T15:16:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.068+0000 7fe33b7fe640 1 -- 172.21.15.46:0/754083747 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe34407afc0 con 0x7fe35416e8c0 2024-04-15T15:16:05.079 INFO:teuthology.orchestra.run.smithi046.stdout:55834574857 2024-04-15T15:16:05.079 INFO:teuthology.orchestra.run.smithi046.stdout:60129542155 2024-04-15T15:16:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.070+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f5d44005850 con 0x7f5d5c0ead00 2024-04-15T15:16:05.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.071+0000 7f5d60ff9640 1 -- 172.21.15.46:0/4046234100 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5d58058bd0 con 0x7f5d5c0ead00 2024-04-15T15:16:05.080 INFO:teuthology.orchestra.run.smithi046.stdout:60129542155 2024-04-15T15:16:05.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d6c06c8f0 msgr2=0x7f4d6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c034000 tx=0x7f4d7c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 msgr2=0x7f4d8c1528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c1528f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400cb20 tx=0x7f4d74007e00 comp rx=0 tx=0).stop 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 shutdown_connections 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c15f9f0 0x7f4d8c159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d8c15ef00 0x7f4d8c1528f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 --2- 172.21.15.46:0/3581990255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d8c15c410 0x7f4d8c0ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 >> 172.21.15.46:0/3581990255 conn(0x7f4d8c0b2c60 msgr2=0x7f4d8c0b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe34006c970 msgr2=0x7fe34006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7fe3596d9640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe34006c970 0x7fe34006ee30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe348006fd0 tx=0x7fe348008040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.072+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 msgr2=0x7fe354094b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 0x7fe354094b30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe34402bda0 tx=0x7fe3440046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 shutdown_connections 2024-04-15T15:16:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7f4d92a35640 1 -- 172.21.15.46:0/3581990255 wait complete. 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 shutdown_connections 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe34006c970 0x7fe34006ee30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe35416ef60 0x7fe35408c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe35416e8c0 0x7fe354094b30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 --2- 172.21.15.46:0/754083747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3540b3260 0x7fe3540cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 >> 172.21.15.46:0/754083747 conn(0x7fe3540b5680 msgr2=0x7fe35416acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 shutdown_connections 2024-04-15T15:16:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.073+0000 7fe3596d9640 1 -- 172.21.15.46:0/754083747 wait complete. 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.075+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 msgr2=0x7f5d4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.075+0000 7f5d69063640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5d640654e0 tx=0x7f5d640747a0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.075+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c0ead00 msgr2=0x7f5d5c0493e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.075+0000 7f5d69063640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c0ead00 0x7f5d5c0493e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5d5800c3c0 tx=0x7f5d5800c880 comp rx=0 tx=0).stop 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.075+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 shutdown_connections 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.075+0000 7f5d69063640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.076+0000 7f5d69063640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d5c0ead00 0x7f5d5c0493e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.076+0000 7f5d69063640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d5c04ffd0 0x7f5d5c048ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.076+0000 7f5d69063640 1 --2- 172.21.15.46:0/4046234100 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d5c038110 0x7f5d5c0dd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.076+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 >> 172.21.15.46:0/4046234100 conn(0x7f5d5c009fb0 msgr2=0x7f5d5c048470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.076+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 shutdown_connections 2024-04-15T15:16:05.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.076+0000 7f5d69063640 1 -- 172.21.15.46:0/4046234100 wait complete. 2024-04-15T15:16:05.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-15T15:16:05.091 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.091 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-04-15T15:16:05.092 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-04-15T15:16:05.093 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.4 2024-04-15T15:16:05.093 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542155 got 60129542155 for osd.6 2024-04-15T15:16:05.095 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-04-15T15:16:05.096 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542155 got 60129542155 for osd.5 2024-04-15T15:16:05.097 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:05.097 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:16:05.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.239+0000 7f1b7240d640 1 Processor -- start 2024-04-15T15:16:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.239+0000 7f1b7240d640 1 -- start start 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b7240d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1b6c09b880 0x7f1b6c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b7240d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b6c0d0da0 0x7f1b6c0d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b7240d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b6c0d05e0 con 0x7f1b6c070940 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b7240d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1b6c09bce0 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b7240d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f1b6c071260 con 0x7f1b6c09b880 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b6c0d0da0 0x7f1b6c0d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1b6c0d0da0 0x7f1b6c0d11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42326/0 (socket says 172.21.15.46:42326) 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 -- 172.21.15.46:0/3068210621 learned_addr learned my addr 172.21.15.46:0/3068210621 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/3068210621 >> v2:172.21.15.78:3300/0 conn(0x7f1b6c09b880 0x7f1b6c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 -- 172.21.15.46:0/3068210621 >> v2:172.21.15.78:3300/0 conn(0x7f1b6c09b880 msgr2=0x7f1b6c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 --2- 172.21.15.46:0/3068210621 >> v2:172.21.15.78:3300/0 conn(0x7f1b6c09b880 0x7f1b6c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 -- 172.21.15.46:0/3068210621 >> v1:172.21.15.78:6789/0 conn(0x7f1b6c070940 legacy=0x7f1b6c160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.240+0000 7f1b709aa640 1 -- 172.21.15.46:0/3068210621 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c15f720 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.241+0000 7f1b709aa640 1 --2- 172.21.15.46:0/3068210621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c0d11d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0734f0 tx=0x7f1b6402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4360c40c7e4c5cd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.241+0000 7f1b6affd640 1 -- 172.21.15.46:0/3068210621 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6402cb20 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.241+0000 7f1b6affd640 1 -- 172.21.15.46:0/3068210621 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b6402ccc0 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.241+0000 7f1b6affd640 1 -- 172.21.15.46:0/3068210621 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b64034610 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.241+0000 7f1b7240d640 1 -- 172.21.15.46:0/3068210621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 msgr2=0x7f1b6c0d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.241+0000 7f1b7240d640 1 --2- 172.21.15.46:0/3068210621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c0d11d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c0734f0 tx=0x7f1b6402bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.242+0000 7f1b7240d640 1 -- 172.21.15.46:0/3068210621 shutdown_connections 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.242+0000 7f1b7240d640 1 --2- 172.21.15.46:0/3068210621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c0d11d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.242+0000 7f1b7240d640 1 --2- 172.21.15.46:0/3068210621 >> v2:172.21.15.78:3300/0 conn(0x7f1b6c09b880 0x7f1b6c070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.242+0000 7f1b7240d640 1 -- 172.21.15.46:0/3068210621 >> 172.21.15.46:0/3068210621 conn(0x7f1b6c0b3f40 msgr2=0x7f1b6c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.242+0000 7f1b7240d640 1 -- 172.21.15.46:0/3068210621 shutdown_connections 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.242+0000 7f1b7240d640 1 -- 172.21.15.46:0/3068210621 wait complete. 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.243+0000 7f1b7240d640 1 Processor -- start 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.243+0000 7f1b7240d640 1 -- start start 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1b6c070940 0x7f1b6c1e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c09b880 0x7f1b6c1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c0d0880 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6c15fce0 con 0x7f1b6c09b880 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6c0d1710 con 0x7f1b6c070940 2024-04-15T15:16:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42342/0 (socket says 172.21.15.46:42342) 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 -- 172.21.15.46:0/1762024715 learned_addr learned my addr 172.21.15.46:0/1762024715 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b6bfff640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1b6c070940 0x7f1b6c1e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b6b7fe640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c09b880 0x7f1b6c1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 -- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1b6c070940 msgr2=0x7f1b6c1e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1b6c070940 0x7f1b6c1e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 -- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c09b880 msgr2=0x7f1b6c1523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c09b880 0x7f1b6c1523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 -- 172.21.15.46:0/1762024715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6c0b6f50 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b709aa640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c152900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1b64036040 tx=0x7f1b64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b64043020 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b6403e450 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b6c0b7230 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.244+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b64009510 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.245+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b6c0b77f0 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.246+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b6c0689e0 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.246+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b64046020 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.247+0000 7f1b68ff9640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b5806c8f0 0x7f1b5806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.247+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1b640ae4a0 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.247+0000 7f1b6bfff640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b5806c8f0 0x7f1b5806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.247+0000 7f1b6bfff640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b5806c8f0 0x7f1b5806edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c001780 tx=0x7f1b5c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.251+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b6404d030 con 0x7f1b6c0d0da0 2024-04-15T15:16:05.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.436+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b6c070720 con 0x7f1b5806c8f0 2024-04-15T15:16:05.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.445+0000 7f1b68ff9640 1 -- 172.21.15.46:0/1762024715 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7f1b6c070720 con 0x7f1b5806c8f0 2024-04-15T15:16:05.450 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:05.452 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:16:05.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b5806c8f0 msgr2=0x7f1b5806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b5806c8f0 0x7f1b5806edb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c001780 tx=0x7f1b5c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 msgr2=0x7f1b6c152900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c152900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1b64036040 tx=0x7f1b64004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:05.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 shutdown_connections 2024-04-15T15:16:05.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b5806c8f0 0x7f1b5806edb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b6c0d0da0 0x7f1b6c152900 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1b6c09b880 0x7f1b6c1523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 --2- 172.21.15.46:0/1762024715 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1b6c070940 0x7f1b6c1e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 >> 172.21.15.46:0/1762024715 conn(0x7f1b6c0b3f40 msgr2=0x7f1b6c15e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 shutdown_connections 2024-04-15T15:16:05.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.448+0000 7f1b7240d640 1 -- 172.21.15.46:0/1762024715 wait complete. 2024-04-15T15:16:05.462 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-15T15:16:04.359458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231928,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742792,"statfs":{"total":773094113280,"available":772856619008,"internally_reserved":0,"allocated":3350528,"data_stored":2022712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439492+0000","last_change":"2024-04-15T15:15:37.566709+0000","last_active":"2024-04-15T15:15:44.439492+0000","last_peered":"2024-04-15T15:15:44.439492+0000","last_clean":"2024-04-15T15:15:44.439492+0000","last_became_active":"2024-04-15T15:15:37.566073+0000","last_became_peered":"2024-04-15T15:15:37.566073+0000","last_unstale":"2024-04-15T15:15:44.439492+0000","last_undegraded":"2024-04-15T15:15:44.439492+0000","last_fullsized":"2024-04-15T15:15:44.439492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:40:55.907098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440303+0000","last_change":"2024-04-15T15:15:38.565494+0000","last_active":"2024-04-15T15:15:44.440303+0000","last_peered":"2024-04-15T15:15:44.440303+0000","last_clean":"2024-04-15T15:15:44.440303+0000","last_became_active":"2024-04-15T15:15:38.563911+0000","last_became_peered":"2024-04-15T15:15:38.563911+0000","last_unstale":"2024-04-15T15:15:44.440303+0000","last_undegraded":"2024-04-15T15:15:44.440303+0000","last_fullsized":"2024-04-15T15:15:44.440303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439886+0000","last_change":"2024-04-15T15:15:42.661119+0000","last_active":"2024-04-15T15:15:44.439886+0000","last_peered":"2024-04-15T15:15:44.439886+0000","last_clean":"2024-04-15T15:15:44.439886+0000","last_became_active":"2024-04-15T15:15:42.660440+0000","last_became_peered":"2024-04-15T15:15:42.660440+0000","last_unstale":"2024-04-15T15:15:44.439886+0000","last_undegraded":"2024-04-15T15:15:44.439886+0000","last_fullsized":"2024-04-15T15:15:44.439886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440400+0000","last_change":"2024-04-15T15:15:37.563320+0000","last_active":"2024-04-15T15:15:44.440400+0000","last_peered":"2024-04-15T15:15:44.440400+0000","last_clean":"2024-04-15T15:15:44.440400+0000","last_became_active":"2024-04-15T15:15:37.563058+0000","last_became_peered":"2024-04-15T15:15:37.563058+0000","last_unstale":"2024-04-15T15:15:44.440400+0000","last_undegraded":"2024-04-15T15:15:44.440400+0000","last_fullsized":"2024-04-15T15:15:44.440400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439174+0000","last_change":"2024-04-15T15:15:38.573022+0000","last_active":"2024-04-15T15:15:44.439174+0000","last_peered":"2024-04-15T15:15:44.439174+0000","last_clean":"2024-04-15T15:15:44.439174+0000","last_became_active":"2024-04-15T15:15:38.572726+0000","last_became_peered":"2024-04-15T15:15:38.572726+0000","last_unstale":"2024-04-15T15:15:44.439174+0000","last_undegraded":"2024-04-15T15:15:44.439174+0000","last_fullsized":"2024-04-15T15:15:44.439174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587093+0000","last_change":"2024-04-15T15:15:42.590697+0000","last_active":"2024-04-15T15:15:43.587093+0000","last_peered":"2024-04-15T15:15:43.587093+0000","last_clean":"2024-04-15T15:15:43.587093+0000","last_became_active":"2024-04-15T15:15:42.590272+0000","last_became_peered":"2024-04-15T15:15:42.590272+0000","last_unstale":"2024-04-15T15:15:43.587093+0000","last_undegraded":"2024-04-15T15:15:43.587093+0000","last_fullsized":"2024-04-15T15:15:43.587093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:10.918812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439863+0000","last_change":"2024-04-15T15:15:37.564953+0000","last_active":"2024-04-15T15:15:44.439863+0000","last_peered":"2024-04-15T15:15:44.439863+0000","last_clean":"2024-04-15T15:15:44.439863+0000","last_became_active":"2024-04-15T15:15:37.564653+0000","last_became_peered":"2024-04-15T15:15:37.564653+0000","last_unstale":"2024-04-15T15:15:44.439863+0000","last_undegraded":"2024-04-15T15:15:44.439863+0000","last_fullsized":"2024-04-15T15:15:44.439863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440820+0000","last_change":"2024-04-15T15:15:38.568986+0000","last_active":"2024-04-15T15:15:44.440820+0000","last_peered":"2024-04-15T15:15:44.440820+0000","last_clean":"2024-04-15T15:15:44.440820+0000","last_became_active":"2024-04-15T15:15:38.566559+0000","last_became_peered":"2024-04-15T15:15:38.566559+0000","last_unstale":"2024-04-15T15:15:44.440820+0000","last_undegraded":"2024-04-15T15:15:44.440820+0000","last_fullsized":"2024-04-15T15:15:44.440820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587815+0000","last_change":"2024-04-15T15:15:42.592706+0000","last_active":"2024-04-15T15:15:43.587815+0000","last_peered":"2024-04-15T15:15:43.587815+0000","last_clean":"2024-04-15T15:15:43.587815+0000","last_became_active":"2024-04-15T15:15:42.592134+0000","last_became_peered":"2024-04-15T15:15:42.592134+0000","last_unstale":"2024-04-15T15:15:43.587815+0000","last_undegraded":"2024-04-15T15:15:43.587815+0000","last_fullsized":"2024-04-15T15:15:43.587815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:41.894568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439525+0000","last_change":"2024-04-15T15:15:37.567857+0000","last_active":"2024-04-15T15:15:44.439525+0000","last_peered":"2024-04-15T15:15:44.439525+0000","last_clean":"2024-04-15T15:15:44.439525+0000","last_became_active":"2024-04-15T15:15:37.567560+0000","last_became_peered":"2024-04-15T15:15:37.567560+0000","last_unstale":"2024-04-15T15:15:44.439525+0000","last_undegraded":"2024-04-15T15:15:44.439525+0000","last_fullsized":"2024-04-15T15:15:44.439525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:36:21.001555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440328+0000","last_change":"2024-04-15T15:15:38.568788+0000","last_active":"2024-04-15T15:15:44.440328+0000","last_peered":"2024-04-15T15:15:44.440328+0000","last_clean":"2024-04-15T15:15:44.440328+0000","last_became_active":"2024-04-15T15:15:38.567501+0000","last_became_peered":"2024-04-15T15:15:38.567501+0000","last_unstale":"2024-04-15T15:15:44.440328+0000","last_undegraded":"2024-04-15T15:15:44.440328+0000","last_fullsized":"2024-04-15T15:15:44.440328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587164+0000","last_change":"2024-04-15T15:15:42.587928+0000","last_active":"2024-04-15T15:15:43.587164+0000","last_peered":"2024-04-15T15:15:43.587164+0000","last_clean":"2024-04-15T15:15:43.587164+0000","last_became_active":"2024-04-15T15:15:42.587518+0000","last_became_peered":"2024-04-15T15:15:42.587518+0000","last_unstale":"2024-04-15T15:15:43.587164+0000","last_undegraded":"2024-04-15T15:15:43.587164+0000","last_fullsized":"2024-04-15T15:15:43.587164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:50.908590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587148+0000","last_change":"2024-04-15T15:15:38.560568+0000","last_active":"2024-04-15T15:15:43.587148+0000","last_peered":"2024-04-15T15:15:43.587148+0000","last_clean":"2024-04-15T15:15:43.587148+0000","last_became_active":"2024-04-15T15:15:38.560292+0000","last_became_peered":"2024-04-15T15:15:38.560292+0000","last_unstale":"2024-04-15T15:15:43.587148+0000","last_undegraded":"2024-04-15T15:15:43.587148+0000","last_fullsized":"2024-04-15T15:15:43.587148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440210+0000","last_change":"2024-04-15T15:15:37.570166+0000","last_active":"2024-04-15T15:15:44.440210+0000","last_peered":"2024-04-15T15:15:44.440210+0000","last_clean":"2024-04-15T15:15:44.440210+0000","last_became_active":"2024-04-15T15:15:37.569862+0000","last_became_peered":"2024-04-15T15:15:37.569862+0000","last_unstale":"2024-04-15T15:15:44.440210+0000","last_undegraded":"2024-04-15T15:15:44.440210+0000","last_fullsized":"2024-04-15T15:15:44.440210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:55:13.182192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588794+0000","last_change":"2024-04-15T15:15:42.590929+0000","last_active":"2024-04-15T15:15:43.588794+0000","last_peered":"2024-04-15T15:15:43.588794+0000","last_clean":"2024-04-15T15:15:43.588794+0000","last_became_active":"2024-04-15T15:15:42.589759+0000","last_became_peered":"2024-04-15T15:15:42.589759+0000","last_unstale":"2024-04-15T15:15:43.588794+0000","last_undegraded":"2024-04-15T15:15:43.588794+0000","last_fullsized":"2024-04-15T15:15:43.588794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439955+0000","last_change":"2024-04-15T15:15:38.563606+0000","last_active":"2024-04-15T15:15:44.439955+0000","last_peered":"2024-04-15T15:15:44.439955+0000","last_clean":"2024-04-15T15:15:44.439955+0000","last_became_active":"2024-04-15T15:15:38.562885+0000","last_became_peered":"2024-04-15T15:15:38.562885+0000","last_unstale":"2024-04-15T15:15:44.439955+0000","last_undegraded":"2024-04-15T15:15:44.439955+0000","last_fullsized":"2024-04-15T15:15:44.439955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587327+0000","last_change":"2024-04-15T15:15:37.562063+0000","last_active":"2024-04-15T15:15:43.587327+0000","last_peered":"2024-04-15T15:15:43.587327+0000","last_clean":"2024-04-15T15:15:43.587327+0000","last_became_active":"2024-04-15T15:15:37.561757+0000","last_became_peered":"2024-04-15T15:15:37.561757+0000","last_unstale":"2024-04-15T15:15:43.587327+0000","last_undegraded":"2024-04-15T15:15:43.587327+0000","last_fullsized":"2024-04-15T15:15:43.587327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588354+0000","last_change":"2024-04-15T15:15:42.591439+0000","last_active":"2024-04-15T15:15:43.588354+0000","last_peered":"2024-04-15T15:15:43.588354+0000","last_clean":"2024-04-15T15:15:43.588354+0000","last_became_active":"2024-04-15T15:15:42.591030+0000","last_became_peered":"2024-04-15T15:15:42.591030+0000","last_unstale":"2024-04-15T15:15:43.588354+0000","last_undegraded":"2024-04-15T15:15:43.588354+0000","last_fullsized":"2024-04-15T15:15:43.588354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:19.368841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439666+0000","last_change":"2024-04-15T15:15:38.566770+0000","last_active":"2024-04-15T15:15:44.439666+0000","last_peered":"2024-04-15T15:15:44.439666+0000","last_clean":"2024-04-15T15:15:44.439666+0000","last_became_active":"2024-04-15T15:15:38.566484+0000","last_became_peered":"2024-04-15T15:15:38.566484+0000","last_unstale":"2024-04-15T15:15:44.439666+0000","last_undegraded":"2024-04-15T15:15:44.439666+0000","last_fullsized":"2024-04-15T15:15:44.439666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586733+0000","last_change":"2024-04-15T15:15:37.559565+0000","last_active":"2024-04-15T15:15:43.586733+0000","last_peered":"2024-04-15T15:15:43.586733+0000","last_clean":"2024-04-15T15:15:43.586733+0000","last_became_active":"2024-04-15T15:15:37.558121+0000","last_became_peered":"2024-04-15T15:15:37.558121+0000","last_unstale":"2024-04-15T15:15:43.586733+0000","last_undegraded":"2024-04-15T15:15:43.586733+0000","last_fullsized":"2024-04-15T15:15:43.586733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.655888+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.655425+0000","last_became_peered":"2024-04-15T15:15:42.655425+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587565+0000","last_change":"2024-04-15T15:15:38.558135+0000","last_active":"2024-04-15T15:15:43.587565+0000","last_peered":"2024-04-15T15:15:43.587565+0000","last_clean":"2024-04-15T15:15:43.587565+0000","last_became_active":"2024-04-15T15:15:38.557849+0000","last_became_peered":"2024-04-15T15:15:38.557849+0000","last_unstale":"2024-04-15T15:15:43.587565+0000","last_undegraded":"2024-04-15T15:15:43.587565+0000","last_fullsized":"2024-04-15T15:15:43.587565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:08.649907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439485+0000","last_change":"2024-04-15T15:15:37.568252+0000","last_active":"2024-04-15T15:15:44.439485+0000","last_peered":"2024-04-15T15:15:44.439485+0000","last_clean":"2024-04-15T15:15:44.439485+0000","last_became_active":"2024-04-15T15:15:37.567982+0000","last_became_peered":"2024-04-15T15:15:37.567982+0000","last_unstale":"2024-04-15T15:15:44.439485+0000","last_undegraded":"2024-04-15T15:15:44.439485+0000","last_fullsized":"2024-04-15T15:15:44.439485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588452+0000","last_change":"2024-04-15T15:15:42.593489+0000","last_active":"2024-04-15T15:15:43.588452+0000","last_peered":"2024-04-15T15:15:43.588452+0000","last_clean":"2024-04-15T15:15:43.588452+0000","last_became_active":"2024-04-15T15:15:42.593152+0000","last_became_peered":"2024-04-15T15:15:42.593152+0000","last_unstale":"2024-04-15T15:15:43.588452+0000","last_undegraded":"2024-04-15T15:15:43.588452+0000","last_fullsized":"2024-04-15T15:15:43.588452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439610+0000","last_change":"2024-04-15T15:15:38.559585+0000","last_active":"2024-04-15T15:15:44.439610+0000","last_peered":"2024-04-15T15:15:44.439610+0000","last_clean":"2024-04-15T15:15:44.439610+0000","last_became_active":"2024-04-15T15:15:38.559225+0000","last_became_peered":"2024-04-15T15:15:38.559225+0000","last_unstale":"2024-04-15T15:15:44.439610+0000","last_undegraded":"2024-04-15T15:15:44.439610+0000","last_fullsized":"2024-04-15T15:15:44.439610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.589097+0000","last_change":"2024-04-15T15:15:37.558650+0000","last_active":"2024-04-15T15:15:43.589097+0000","last_peered":"2024-04-15T15:15:43.589097+0000","last_clean":"2024-04-15T15:15:43.589097+0000","last_became_active":"2024-04-15T15:15:37.558128+0000","last_became_peered":"2024-04-15T15:15:37.558128+0000","last_unstale":"2024-04-15T15:15:43.589097+0000","last_undegraded":"2024-04-15T15:15:43.589097+0000","last_fullsized":"2024-04-15T15:15:43.589097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587388+0000","last_change":"2024-04-15T15:15:42.655806+0000","last_active":"2024-04-15T15:15:43.587388+0000","last_peered":"2024-04-15T15:15:43.587388+0000","last_clean":"2024-04-15T15:15:43.587388+0000","last_became_active":"2024-04-15T15:15:42.655431+0000","last_became_peered":"2024-04-15T15:15:42.655431+0000","last_unstale":"2024-04-15T15:15:43.587388+0000","last_undegraded":"2024-04-15T15:15:43.587388+0000","last_fullsized":"2024-04-15T15:15:43.587388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:10.032080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440420+0000","last_change":"2024-04-15T15:15:38.567366+0000","last_active":"2024-04-15T15:15:44.440420+0000","last_peered":"2024-04-15T15:15:44.440420+0000","last_clean":"2024-04-15T15:15:44.440420+0000","last_became_active":"2024-04-15T15:15:38.567064+0000","last_became_peered":"2024-04-15T15:15:38.567064+0000","last_unstale":"2024-04-15T15:15:44.440420+0000","last_undegraded":"2024-04-15T15:15:44.440420+0000","last_fullsized":"2024-04-15T15:15:44.440420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587807+0000","last_change":"2024-04-15T15:15:37.554351+0000","last_active":"2024-04-15T15:15:43.587807+0000","last_peered":"2024-04-15T15:15:43.587807+0000","last_clean":"2024-04-15T15:15:43.587807+0000","last_became_active":"2024-04-15T15:15:37.554062+0000","last_became_peered":"2024-04-15T15:15:37.554062+0000","last_unstale":"2024-04-15T15:15:43.587807+0000","last_undegraded":"2024-04-15T15:15:43.587807+0000","last_fullsized":"2024-04-15T15:15:43.587807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438909+0000","last_change":"2024-04-15T15:15:42.659938+0000","last_active":"2024-04-15T15:15:44.438909+0000","last_peered":"2024-04-15T15:15:44.438909+0000","last_clean":"2024-04-15T15:15:44.438909+0000","last_became_active":"2024-04-15T15:15:42.659252+0000","last_became_peered":"2024-04-15T15:15:42.659252+0000","last_unstale":"2024-04-15T15:15:44.438909+0000","last_undegraded":"2024-04-15T15:15:44.438909+0000","last_fullsized":"2024-04-15T15:15:44.438909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:48.210859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587105+0000","last_change":"2024-04-15T15:15:38.561075+0000","last_active":"2024-04-15T15:15:43.587105+0000","last_peered":"2024-04-15T15:15:43.587105+0000","last_clean":"2024-04-15T15:15:43.587105+0000","last_became_active":"2024-04-15T15:15:38.559163+0000","last_became_peered":"2024-04-15T15:15:38.559163+0000","last_unstale":"2024-04-15T15:15:43.587105+0000","last_undegraded":"2024-04-15T15:15:43.587105+0000","last_fullsized":"2024-04-15T15:15:43.587105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440117+0000","last_change":"2024-04-15T15:15:37.568261+0000","last_active":"2024-04-15T15:15:44.440117+0000","last_peered":"2024-04-15T15:15:44.440117+0000","last_clean":"2024-04-15T15:15:44.440117+0000","last_became_active":"2024-04-15T15:15:37.567997+0000","last_became_peered":"2024-04-15T15:15:37.567997+0000","last_unstale":"2024-04-15T15:15:44.440117+0000","last_undegraded":"2024-04-15T15:15:44.440117+0000","last_fullsized":"2024-04-15T15:15:44.440117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:47:24.543446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:42.658160+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:42.657766+0000","last_became_peered":"2024-04-15T15:15:42.657766+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440434+0000","last_change":"2024-04-15T15:15:38.563836+0000","last_active":"2024-04-15T15:15:44.440434+0000","last_peered":"2024-04-15T15:15:44.440434+0000","last_clean":"2024-04-15T15:15:44.440434+0000","last_became_active":"2024-04-15T15:15:38.563487+0000","last_became_peered":"2024-04-15T15:15:38.563487+0000","last_unstale":"2024-04-15T15:15:44.440434+0000","last_undegraded":"2024-04-15T15:15:44.440434+0000","last_fullsized":"2024-04-15T15:15:44.440434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587767+0000","last_change":"2024-04-15T15:15:37.563786+0000","last_active":"2024-04-15T15:15:43.587767+0000","last_peered":"2024-04-15T15:15:43.587767+0000","last_clean":"2024-04-15T15:15:43.587767+0000","last_became_active":"2024-04-15T15:15:37.563491+0000","last_became_peered":"2024-04-15T15:15:37.563491+0000","last_unstale":"2024-04-15T15:15:43.587767+0000","last_undegraded":"2024-04-15T15:15:43.587767+0000","last_fullsized":"2024-04-15T15:15:43.587767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586975+0000","last_change":"2024-04-15T15:15:42.593439+0000","last_active":"2024-04-15T15:15:43.586975+0000","last_peered":"2024-04-15T15:15:43.586975+0000","last_clean":"2024-04-15T15:15:43.586975+0000","last_became_active":"2024-04-15T15:15:42.591555+0000","last_became_peered":"2024-04-15T15:15:42.591555+0000","last_unstale":"2024-04-15T15:15:43.586975+0000","last_undegraded":"2024-04-15T15:15:43.586975+0000","last_fullsized":"2024-04-15T15:15:43.586975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588422+0000","last_change":"2024-04-15T15:15:38.560053+0000","last_active":"2024-04-15T15:15:43.588422+0000","last_peered":"2024-04-15T15:15:43.588422+0000","last_clean":"2024-04-15T15:15:43.588422+0000","last_became_active":"2024-04-15T15:15:38.559462+0000","last_became_peered":"2024-04-15T15:15:38.559462+0000","last_unstale":"2024-04-15T15:15:43.588422+0000","last_undegraded":"2024-04-15T15:15:43.588422+0000","last_fullsized":"2024-04-15T15:15:43.588422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:50.427117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440424+0000","last_change":"2024-04-15T15:15:37.561463+0000","last_active":"2024-04-15T15:15:44.440424+0000","last_peered":"2024-04-15T15:15:44.440424+0000","last_clean":"2024-04-15T15:15:44.440424+0000","last_became_active":"2024-04-15T15:15:37.560849+0000","last_became_peered":"2024-04-15T15:15:37.560849+0000","last_unstale":"2024-04-15T15:15:44.440424+0000","last_undegraded":"2024-04-15T15:15:44.440424+0000","last_fullsized":"2024-04-15T15:15:44.440424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.594805+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.594322+0000","last_became_peered":"2024-04-15T15:15:42.594322+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586925+0000","last_change":"2024-04-15T15:15:38.559721+0000","last_active":"2024-04-15T15:15:43.586925+0000","last_peered":"2024-04-15T15:15:43.586925+0000","last_clean":"2024-04-15T15:15:43.586925+0000","last_became_active":"2024-04-15T15:15:38.559440+0000","last_became_peered":"2024-04-15T15:15:38.559440+0000","last_unstale":"2024-04-15T15:15:43.586925+0000","last_undegraded":"2024-04-15T15:15:43.586925+0000","last_fullsized":"2024-04-15T15:15:43.586925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769205+0000","last_change":"2024-04-15T15:15:37.562651+0000","last_active":"2024-04-15T15:15:43.769205+0000","last_peered":"2024-04-15T15:15:43.769205+0000","last_clean":"2024-04-15T15:15:43.769205+0000","last_became_active":"2024-04-15T15:15:37.562267+0000","last_became_peered":"2024-04-15T15:15:37.562267+0000","last_unstale":"2024-04-15T15:15:43.769205+0000","last_undegraded":"2024-04-15T15:15:43.769205+0000","last_fullsized":"2024-04-15T15:15:43.769205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587782+0000","last_change":"2024-04-15T15:15:42.595294+0000","last_active":"2024-04-15T15:15:43.587782+0000","last_peered":"2024-04-15T15:15:43.587782+0000","last_clean":"2024-04-15T15:15:43.587782+0000","last_became_active":"2024-04-15T15:15:42.594395+0000","last_became_peered":"2024-04-15T15:15:42.594395+0000","last_unstale":"2024-04-15T15:15:43.587782+0000","last_undegraded":"2024-04-15T15:15:43.587782+0000","last_fullsized":"2024-04-15T15:15:43.587782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438893+0000","last_change":"2024-04-15T15:15:38.566568+0000","last_active":"2024-04-15T15:15:44.438893+0000","last_peered":"2024-04-15T15:15:44.438893+0000","last_clean":"2024-04-15T15:15:44.438893+0000","last_became_active":"2024-04-15T15:15:38.566261+0000","last_became_peered":"2024-04-15T15:15:38.566261+0000","last_unstale":"2024-04-15T15:15:44.438893+0000","last_undegraded":"2024-04-15T15:15:44.438893+0000","last_fullsized":"2024-04-15T15:15:44.438893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439388+0000","last_change":"2024-04-15T15:15:37.567337+0000","last_active":"2024-04-15T15:15:44.439388+0000","last_peered":"2024-04-15T15:15:44.439388+0000","last_clean":"2024-04-15T15:15:44.439388+0000","last_became_active":"2024-04-15T15:15:37.567042+0000","last_became_peered":"2024-04-15T15:15:37.567042+0000","last_unstale":"2024-04-15T15:15:44.439388+0000","last_undegraded":"2024-04-15T15:15:44.439388+0000","last_fullsized":"2024-04-15T15:15:44.439388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439562+0000","last_change":"2024-04-15T15:15:42.657091+0000","last_active":"2024-04-15T15:15:44.439562+0000","last_peered":"2024-04-15T15:15:44.439562+0000","last_clean":"2024-04-15T15:15:44.439562+0000","last_became_active":"2024-04-15T15:15:42.656638+0000","last_became_peered":"2024-04-15T15:15:42.656638+0000","last_unstale":"2024-04-15T15:15:44.439562+0000","last_undegraded":"2024-04-15T15:15:44.439562+0000","last_fullsized":"2024-04-15T15:15:44.439562+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:46.336878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439991+0000","last_change":"2024-04-15T15:15:38.576736+0000","last_active":"2024-04-15T15:15:44.439991+0000","last_peered":"2024-04-15T15:15:44.439991+0000","last_clean":"2024-04-15T15:15:44.439991+0000","last_became_active":"2024-04-15T15:15:38.574326+0000","last_became_peered":"2024-04-15T15:15:38.574326+0000","last_unstale":"2024-04-15T15:15:44.439991+0000","last_undegraded":"2024-04-15T15:15:44.439991+0000","last_fullsized":"2024-04-15T15:15:44.439991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440769+0000","last_change":"2024-04-15T15:15:37.557351+0000","last_active":"2024-04-15T15:15:44.440769+0000","last_peered":"2024-04-15T15:15:44.440769+0000","last_clean":"2024-04-15T15:15:44.440769+0000","last_became_active":"2024-04-15T15:15:37.557036+0000","last_became_peered":"2024-04-15T15:15:37.557036+0000","last_unstale":"2024-04-15T15:15:44.440769+0000","last_undegraded":"2024-04-15T15:15:44.440769+0000","last_fullsized":"2024-04-15T15:15:44.440769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439564+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:15:44.439564+0000","last_peered":"2024-04-15T15:15:44.439564+0000","last_clean":"2024-04-15T15:15:44.439564+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:15:44.439564+0000","last_undegraded":"2024-04-15T15:15:44.439564+0000","last_fullsized":"2024-04-15T15:15:44.439564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588337+0000","last_change":"2024-04-15T15:15:42.658428+0000","last_active":"2024-04-15T15:15:43.588337+0000","last_peered":"2024-04-15T15:15:43.588337+0000","last_clean":"2024-04-15T15:15:43.588337+0000","last_became_active":"2024-04-15T15:15:42.658014+0000","last_became_peered":"2024-04-15T15:15:42.658014+0000","last_unstale":"2024-04-15T15:15:43.588337+0000","last_undegraded":"2024-04-15T15:15:43.588337+0000","last_fullsized":"2024-04-15T15:15:43.588337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439749+0000","last_change":"2024-04-15T15:15:38.572859+0000","last_active":"2024-04-15T15:15:44.439749+0000","last_peered":"2024-04-15T15:15:44.439749+0000","last_clean":"2024-04-15T15:15:44.439749+0000","last_became_active":"2024-04-15T15:15:38.572580+0000","last_became_peered":"2024-04-15T15:15:38.572580+0000","last_unstale":"2024-04-15T15:15:44.439749+0000","last_undegraded":"2024-04-15T15:15:44.439749+0000","last_fullsized":"2024-04-15T15:15:44.439749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769429+0000","last_change":"2024-04-15T15:15:37.555795+0000","last_active":"2024-04-15T15:15:43.769429+0000","last_peered":"2024-04-15T15:15:43.769429+0000","last_clean":"2024-04-15T15:15:43.769429+0000","last_became_active":"2024-04-15T15:15:37.555502+0000","last_became_peered":"2024-04-15T15:15:37.555502+0000","last_unstale":"2024-04-15T15:15:43.769429+0000","last_undegraded":"2024-04-15T15:15:43.769429+0000","last_fullsized":"2024-04-15T15:15:43.769429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587412+0000","last_change":"2024-04-15T15:15:42.655842+0000","last_active":"2024-04-15T15:15:43.587412+0000","last_peered":"2024-04-15T15:15:43.587412+0000","last_clean":"2024-04-15T15:15:43.587412+0000","last_became_active":"2024-04-15T15:15:42.655470+0000","last_became_peered":"2024-04-15T15:15:42.655470+0000","last_unstale":"2024-04-15T15:15:43.587412+0000","last_undegraded":"2024-04-15T15:15:43.587412+0000","last_fullsized":"2024-04-15T15:15:43.587412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440874+0000","last_change":"2024-04-15T15:15:38.573853+0000","last_active":"2024-04-15T15:15:44.440874+0000","last_peered":"2024-04-15T15:15:44.440874+0000","last_clean":"2024-04-15T15:15:44.440874+0000","last_became_active":"2024-04-15T15:15:38.573585+0000","last_became_peered":"2024-04-15T15:15:38.573585+0000","last_unstale":"2024-04-15T15:15:44.440874+0000","last_undegraded":"2024-04-15T15:15:44.440874+0000","last_fullsized":"2024-04-15T15:15:44.440874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587697+0000","last_change":"2024-04-15T15:15:37.557027+0000","last_active":"2024-04-15T15:15:43.587697+0000","last_peered":"2024-04-15T15:15:43.587697+0000","last_clean":"2024-04-15T15:15:43.587697+0000","last_became_active":"2024-04-15T15:15:37.556725+0000","last_became_peered":"2024-04-15T15:15:37.556725+0000","last_unstale":"2024-04-15T15:15:43.587697+0000","last_undegraded":"2024-04-15T15:15:43.587697+0000","last_fullsized":"2024-04-15T15:15:43.587697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439555+0000","last_change":"2024-04-15T15:15:42.593049+0000","last_active":"2024-04-15T15:15:44.439555+0000","last_peered":"2024-04-15T15:15:44.439555+0000","last_clean":"2024-04-15T15:15:44.439555+0000","last_became_active":"2024-04-15T15:15:42.592494+0000","last_became_peered":"2024-04-15T15:15:42.592494+0000","last_unstale":"2024-04-15T15:15:44.439555+0000","last_undegraded":"2024-04-15T15:15:44.439555+0000","last_fullsized":"2024-04-15T15:15:44.439555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440121+0000","last_change":"2024-04-15T15:15:38.574063+0000","last_active":"2024-04-15T15:15:44.440121+0000","last_peered":"2024-04-15T15:15:44.440121+0000","last_clean":"2024-04-15T15:15:44.440121+0000","last_became_active":"2024-04-15T15:15:38.572802+0000","last_became_peered":"2024-04-15T15:15:38.572802+0000","last_unstale":"2024-04-15T15:15:44.440121+0000","last_undegraded":"2024-04-15T15:15:44.440121+0000","last_fullsized":"2024-04-15T15:15:44.440121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:42.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769497+0000","last_change":"2024-04-15T15:15:37.563114+0000","last_active":"2024-04-15T15:15:43.769497+0000","last_peered":"2024-04-15T15:15:43.769497+0000","last_clean":"2024-04-15T15:15:43.769497+0000","last_became_active":"2024-04-15T15:15:37.562746+0000","last_became_peered":"2024-04-15T15:15:37.562746+0000","last_unstale":"2024-04-15T15:15:43.769497+0000","last_undegraded":"2024-04-15T15:15:43.769497+0000","last_fullsized":"2024-04-15T15:15:43.769497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439239+0000","last_change":"2024-04-15T15:15:42.658786+0000","last_active":"2024-04-15T15:15:44.439239+0000","last_peered":"2024-04-15T15:15:44.439239+0000","last_clean":"2024-04-15T15:15:44.439239+0000","last_became_active":"2024-04-15T15:15:42.658329+0000","last_became_peered":"2024-04-15T15:15:42.658329+0000","last_unstale":"2024-04-15T15:15:44.439239+0000","last_undegraded":"2024-04-15T15:15:44.439239+0000","last_fullsized":"2024-04-15T15:15:44.439239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439930+0000","last_change":"2024-04-15T15:15:38.573032+0000","last_active":"2024-04-15T15:15:44.439930+0000","last_peered":"2024-04-15T15:15:44.439930+0000","last_clean":"2024-04-15T15:15:44.439930+0000","last_became_active":"2024-04-15T15:15:38.572689+0000","last_became_peered":"2024-04-15T15:15:38.572689+0000","last_unstale":"2024-04-15T15:15:44.439930+0000","last_undegraded":"2024-04-15T15:15:44.439930+0000","last_fullsized":"2024-04-15T15:15:44.439930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588533+0000","last_change":"2024-04-15T15:15:37.565326+0000","last_active":"2024-04-15T15:15:43.588533+0000","last_peered":"2024-04-15T15:15:43.588533+0000","last_clean":"2024-04-15T15:15:43.588533+0000","last_became_active":"2024-04-15T15:15:37.564993+0000","last_became_peered":"2024-04-15T15:15:37.564993+0000","last_unstale":"2024-04-15T15:15:43.588533+0000","last_undegraded":"2024-04-15T15:15:43.588533+0000","last_fullsized":"2024-04-15T15:15:43.588533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:03:33.287332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588553+0000","last_change":"2024-04-15T15:15:42.590642+0000","last_active":"2024-04-15T15:15:43.588553+0000","last_peered":"2024-04-15T15:15:43.588553+0000","last_clean":"2024-04-15T15:15:43.588553+0000","last_became_active":"2024-04-15T15:15:42.590216+0000","last_became_peered":"2024-04-15T15:15:42.590216+0000","last_unstale":"2024-04-15T15:15:43.588553+0000","last_undegraded":"2024-04-15T15:15:43.588553+0000","last_fullsized":"2024-04-15T15:15:43.588553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439674+0000","last_change":"2024-04-15T15:15:38.571884+0000","last_active":"2024-04-15T15:15:44.439674+0000","last_peered":"2024-04-15T15:15:44.439674+0000","last_clean":"2024-04-15T15:15:44.439674+0000","last_became_active":"2024-04-15T15:15:38.571574+0000","last_became_peered":"2024-04-15T15:15:38.571574+0000","last_unstale":"2024-04-15T15:15:44.439674+0000","last_undegraded":"2024-04-15T15:15:44.439674+0000","last_fullsized":"2024-04-15T15:15:44.439674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769091+0000","last_change":"2024-04-15T15:15:37.558157+0000","last_active":"2024-04-15T15:15:43.769091+0000","last_peered":"2024-04-15T15:15:43.769091+0000","last_clean":"2024-04-15T15:15:43.769091+0000","last_became_active":"2024-04-15T15:15:37.557622+0000","last_became_peered":"2024-04-15T15:15:37.557622+0000","last_unstale":"2024-04-15T15:15:43.769091+0000","last_undegraded":"2024-04-15T15:15:43.769091+0000","last_fullsized":"2024-04-15T15:15:43.769091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439409+0000","last_change":"2024-04-15T15:15:42.589002+0000","last_active":"2024-04-15T15:15:44.439409+0000","last_peered":"2024-04-15T15:15:44.439409+0000","last_clean":"2024-04-15T15:15:44.439409+0000","last_became_active":"2024-04-15T15:15:42.588573+0000","last_became_peered":"2024-04-15T15:15:42.588573+0000","last_unstale":"2024-04-15T15:15:44.439409+0000","last_undegraded":"2024-04-15T15:15:44.439409+0000","last_fullsized":"2024-04-15T15:15:44.439409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588824+0000","last_change":"2024-04-15T15:15:38.565226+0000","last_active":"2024-04-15T15:15:43.588824+0000","last_peered":"2024-04-15T15:15:43.588824+0000","last_clean":"2024-04-15T15:15:43.588824+0000","last_became_active":"2024-04-15T15:15:38.564938+0000","last_became_peered":"2024-04-15T15:15:38.564938+0000","last_unstale":"2024-04-15T15:15:43.588824+0000","last_undegraded":"2024-04-15T15:15:43.588824+0000","last_fullsized":"2024-04-15T15:15:43.588824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587872+0000","last_change":"2024-04-15T15:15:37.560743+0000","last_active":"2024-04-15T15:15:43.587872+0000","last_peered":"2024-04-15T15:15:43.587872+0000","last_clean":"2024-04-15T15:15:43.587872+0000","last_became_active":"2024-04-15T15:15:37.560439+0000","last_became_peered":"2024-04-15T15:15:37.560439+0000","last_unstale":"2024-04-15T15:15:43.587872+0000","last_undegraded":"2024-04-15T15:15:43.587872+0000","last_fullsized":"2024-04-15T15:15:43.587872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440270+0000","last_change":"2024-04-15T15:15:42.660245+0000","last_active":"2024-04-15T15:15:44.440270+0000","last_peered":"2024-04-15T15:15:44.440270+0000","last_clean":"2024-04-15T15:15:44.440270+0000","last_became_active":"2024-04-15T15:15:42.659882+0000","last_became_peered":"2024-04-15T15:15:42.659882+0000","last_unstale":"2024-04-15T15:15:44.440270+0000","last_undegraded":"2024-04-15T15:15:44.440270+0000","last_fullsized":"2024-04-15T15:15:44.440270+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:16:55.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440094+0000","last_change":"2024-04-15T15:15:38.572105+0000","last_active":"2024-04-15T15:15:44.440094+0000","last_peered":"2024-04-15T15:15:44.440094+0000","last_clean":"2024-04-15T15:15:44.440094+0000","last_became_active":"2024-04-15T15:15:38.571821+0000","last_became_peered":"2024-04-15T15:15:38.571821+0000","last_unstale":"2024-04-15T15:15:44.440094+0000","last_undegraded":"2024-04-15T15:15:44.440094+0000","last_fullsized":"2024-04-15T15:15:44.440094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440332+0000","last_change":"2024-04-15T15:15:37.562506+0000","last_active":"2024-04-15T15:15:44.440332+0000","last_peered":"2024-04-15T15:15:44.440332+0000","last_clean":"2024-04-15T15:15:44.440332+0000","last_became_active":"2024-04-15T15:15:37.562212+0000","last_became_peered":"2024-04-15T15:15:37.562212+0000","last_unstale":"2024-04-15T15:15:44.440332+0000","last_undegraded":"2024-04-15T15:15:44.440332+0000","last_fullsized":"2024-04-15T15:15:44.440332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440199+0000","last_change":"2024-04-15T15:15:42.656037+0000","last_active":"2024-04-15T15:15:44.440199+0000","last_peered":"2024-04-15T15:15:44.440199+0000","last_clean":"2024-04-15T15:15:44.440199+0000","last_became_active":"2024-04-15T15:15:42.655647+0000","last_became_peered":"2024-04-15T15:15:42.655647+0000","last_unstale":"2024-04-15T15:15:44.440199+0000","last_undegraded":"2024-04-15T15:15:44.440199+0000","last_fullsized":"2024-04-15T15:15:44.440199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:53:58.204060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439284+0000","last_change":"2024-04-15T15:15:38.573064+0000","last_active":"2024-04-15T15:15:44.439284+0000","last_peered":"2024-04-15T15:15:44.439284+0000","last_clean":"2024-04-15T15:15:44.439284+0000","last_became_active":"2024-04-15T15:15:38.572766+0000","last_became_peered":"2024-04-15T15:15:38.572766+0000","last_unstale":"2024-04-15T15:15:44.439284+0000","last_undegraded":"2024-04-15T15:15:44.439284+0000","last_fullsized":"2024-04-15T15:15:44.439284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:53:30.455027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439611+0000","last_change":"2024-04-15T15:15:37.565383+0000","last_active":"2024-04-15T15:15:44.439611+0000","last_peered":"2024-04-15T15:15:44.439611+0000","last_clean":"2024-04-15T15:15:44.439611+0000","last_became_active":"2024-04-15T15:15:37.561007+0000","last_became_peered":"2024-04-15T15:15:37.561007+0000","last_unstale":"2024-04-15T15:15:44.439611+0000","last_undegraded":"2024-04-15T15:15:44.439611+0000","last_fullsized":"2024-04-15T15:15:44.439611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440266+0000","last_change":"2024-04-15T15:15:42.657177+0000","last_active":"2024-04-15T15:15:44.440266+0000","last_peered":"2024-04-15T15:15:44.440266+0000","last_clean":"2024-04-15T15:15:44.440266+0000","last_became_active":"2024-04-15T15:15:42.656776+0000","last_became_peered":"2024-04-15T15:15:42.656776+0000","last_unstale":"2024-04-15T15:15:44.440266+0000","last_undegraded":"2024-04-15T15:15:44.440266+0000","last_fullsized":"2024-04-15T15:15:44.440266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439546+0000","last_change":"2024-04-15T15:15:38.569532+0000","last_active":"2024-04-15T15:15:44.439546+0000","last_peered":"2024-04-15T15:15:44.439546+0000","last_clean":"2024-04-15T15:15:44.439546+0000","last_became_active":"2024-04-15T15:15:38.569097+0000","last_became_peered":"2024-04-15T15:15:38.569097+0000","last_unstale":"2024-04-15T15:15:44.439546+0000","last_undegraded":"2024-04-15T15:15:44.439546+0000","last_fullsized":"2024-04-15T15:15:44.439546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:48.573127+0000","last_change":"2024-04-15T15:15:37.559420+0000","last_active":"2024-04-15T15:15:48.573127+0000","last_peered":"2024-04-15T15:15:48.573127+0000","last_clean":"2024-04-15T15:15:48.573127+0000","last_became_active":"2024-04-15T15:15:37.559152+0000","last_became_peered":"2024-04-15T15:15:37.559152+0000","last_unstale":"2024-04-15T15:15:48.573127+0000","last_undegraded":"2024-04-15T15:15:48.573127+0000","last_fullsized":"2024-04-15T15:15:48.573127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587812+0000","last_change":"2024-04-15T15:15:42.591766+0000","last_active":"2024-04-15T15:15:43.587812+0000","last_peered":"2024-04-15T15:15:43.587812+0000","last_clean":"2024-04-15T15:15:43.587812+0000","last_became_active":"2024-04-15T15:15:42.591354+0000","last_became_peered":"2024-04-15T15:15:42.591354+0000","last_unstale":"2024-04-15T15:15:43.587812+0000","last_undegraded":"2024-04-15T15:15:43.587812+0000","last_fullsized":"2024-04-15T15:15:43.587812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588450+0000","last_change":"2024-04-15T15:15:38.562820+0000","last_active":"2024-04-15T15:15:43.588450+0000","last_peered":"2024-04-15T15:15:43.588450+0000","last_clean":"2024-04-15T15:15:43.588450+0000","last_became_active":"2024-04-15T15:15:38.562536+0000","last_became_peered":"2024-04-15T15:15:38.562536+0000","last_unstale":"2024-04-15T15:15:43.588450+0000","last_undegraded":"2024-04-15T15:15:43.588450+0000","last_fullsized":"2024-04-15T15:15:43.588450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587577+0000","last_change":"2024-04-15T15:15:37.558919+0000","last_active":"2024-04-15T15:15:43.587577+0000","last_peered":"2024-04-15T15:15:43.587577+0000","last_clean":"2024-04-15T15:15:43.587577+0000","last_became_active":"2024-04-15T15:15:37.558592+0000","last_became_peered":"2024-04-15T15:15:37.558592+0000","last_unstale":"2024-04-15T15:15:43.587577+0000","last_undegraded":"2024-04-15T15:15:43.587577+0000","last_fullsized":"2024-04-15T15:15:43.587577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587908+0000","last_change":"2024-04-15T15:15:42.591079+0000","last_active":"2024-04-15T15:15:43.587908+0000","last_peered":"2024-04-15T15:15:43.587908+0000","last_clean":"2024-04-15T15:15:43.587908+0000","last_became_active":"2024-04-15T15:15:42.590633+0000","last_became_peered":"2024-04-15T15:15:42.590633+0000","last_unstale":"2024-04-15T15:15:43.587908+0000","last_undegraded":"2024-04-15T15:15:43.587908+0000","last_fullsized":"2024-04-15T15:15:43.587908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:33.636220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439372+0000","last_change":"2024-04-15T15:15:38.574069+0000","last_active":"2024-04-15T15:15:44.439372+0000","last_peered":"2024-04-15T15:15:44.439372+0000","last_clean":"2024-04-15T15:15:44.439372+0000","last_became_active":"2024-04-15T15:15:38.573139+0000","last_became_peered":"2024-04-15T15:15:38.573139+0000","last_unstale":"2024-04-15T15:15:44.439372+0000","last_undegraded":"2024-04-15T15:15:44.439372+0000","last_fullsized":"2024-04-15T15:15:44.439372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440496+0000","last_change":"2024-04-15T15:15:37.563424+0000","last_active":"2024-04-15T15:15:44.440496+0000","last_peered":"2024-04-15T15:15:44.440496+0000","last_clean":"2024-04-15T15:15:44.440496+0000","last_became_active":"2024-04-15T15:15:37.562924+0000","last_became_peered":"2024-04-15T15:15:37.562924+0000","last_unstale":"2024-04-15T15:15:44.440496+0000","last_undegraded":"2024-04-15T15:15:44.440496+0000","last_fullsized":"2024-04-15T15:15:44.440496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439129+0000","last_change":"2024-04-15T15:15:42.587632+0000","last_active":"2024-04-15T15:15:44.439129+0000","last_peered":"2024-04-15T15:15:44.439129+0000","last_clean":"2024-04-15T15:15:44.439129+0000","last_became_active":"2024-04-15T15:15:42.587139+0000","last_became_peered":"2024-04-15T15:15:42.587139+0000","last_unstale":"2024-04-15T15:15:44.439129+0000","last_undegraded":"2024-04-15T15:15:44.439129+0000","last_fullsized":"2024-04-15T15:15:44.439129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440397+0000","last_change":"2024-04-15T15:15:38.567493+0000","last_active":"2024-04-15T15:15:44.440397+0000","last_peered":"2024-04-15T15:15:44.440397+0000","last_clean":"2024-04-15T15:15:44.440397+0000","last_became_active":"2024-04-15T15:15:38.567188+0000","last_became_peered":"2024-04-15T15:15:38.567188+0000","last_unstale":"2024-04-15T15:15:44.440397+0000","last_undegraded":"2024-04-15T15:15:44.440397+0000","last_fullsized":"2024-04-15T15:15:44.440397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588898+0000","last_change":"2024-04-15T15:15:37.563986+0000","last_active":"2024-04-15T15:15:43.588898+0000","last_peered":"2024-04-15T15:15:43.588898+0000","last_clean":"2024-04-15T15:15:43.588898+0000","last_became_active":"2024-04-15T15:15:37.563666+0000","last_became_peered":"2024-04-15T15:15:37.563666+0000","last_unstale":"2024-04-15T15:15:43.588898+0000","last_undegraded":"2024-04-15T15:15:43.588898+0000","last_fullsized":"2024-04-15T15:15:43.588898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440053+0000","last_change":"2024-04-15T15:15:42.661474+0000","last_active":"2024-04-15T15:15:44.440053+0000","last_peered":"2024-04-15T15:15:44.440053+0000","last_clean":"2024-04-15T15:15:44.440053+0000","last_became_active":"2024-04-15T15:15:42.660740+0000","last_became_peered":"2024-04-15T15:15:42.660740+0000","last_unstale":"2024-04-15T15:15:44.440053+0000","last_undegraded":"2024-04-15T15:15:44.440053+0000","last_fullsized":"2024-04-15T15:15:44.440053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:56:48.355956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439422+0000","last_change":"2024-04-15T15:15:42.658445+0000","last_active":"2024-04-15T15:15:44.439422+0000","last_peered":"2024-04-15T15:15:44.439422+0000","last_clean":"2024-04-15T15:15:44.439422+0000","last_became_active":"2024-04-15T15:15:42.658104+0000","last_became_peered":"2024-04-15T15:15:42.658104+0000","last_unstale":"2024-04-15T15:15:44.439422+0000","last_undegraded":"2024-04-15T15:15:44.439422+0000","last_fullsized":"2024-04-15T15:15:44.439422+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586589+0000","last_change":"2024-04-15T15:15:38.562893+0000","last_active":"2024-04-15T15:15:43.586589+0000","last_peered":"2024-04-15T15:15:43.586589+0000","last_clean":"2024-04-15T15:15:43.586589+0000","last_became_active":"2024-04-15T15:15:38.562604+0000","last_became_peered":"2024-04-15T15:15:38.562604+0000","last_unstale":"2024-04-15T15:15:43.586589+0000","last_undegraded":"2024-04-15T15:15:43.586589+0000","last_fullsized":"2024-04-15T15:15:43.586589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:37.556524+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:37.556249+0000","last_became_peered":"2024-04-15T15:15:37.556249+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588777+0000","last_change":"2024-04-15T15:15:42.590110+0000","last_active":"2024-04-15T15:15:43.588777+0000","last_peered":"2024-04-15T15:15:43.588777+0000","last_clean":"2024-04-15T15:15:43.588777+0000","last_became_active":"2024-04-15T15:15:42.589355+0000","last_became_peered":"2024-04-15T15:15:42.589355+0000","last_unstale":"2024-04-15T15:15:43.588777+0000","last_undegraded":"2024-04-15T15:15:43.588777+0000","last_fullsized":"2024-04-15T15:15:43.588777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439180+0000","last_change":"2024-04-15T15:15:38.567772+0000","last_active":"2024-04-15T15:15:44.439180+0000","last_peered":"2024-04-15T15:15:44.439180+0000","last_clean":"2024-04-15T15:15:44.439180+0000","last_became_active":"2024-04-15T15:15:38.566987+0000","last_became_peered":"2024-04-15T15:15:38.566987+0000","last_unstale":"2024-04-15T15:15:44.439180+0000","last_undegraded":"2024-04-15T15:15:44.439180+0000","last_fullsized":"2024-04-15T15:15:44.439180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440394+0000","last_change":"2024-04-15T15:15:37.561996+0000","last_active":"2024-04-15T15:15:44.440394+0000","last_peered":"2024-04-15T15:15:44.440394+0000","last_clean":"2024-04-15T15:15:44.440394+0000","last_became_active":"2024-04-15T15:15:37.561696+0000","last_became_peered":"2024-04-15T15:15:37.561696+0000","last_unstale":"2024-04-15T15:15:44.440394+0000","last_undegraded":"2024-04-15T15:15:44.440394+0000","last_fullsized":"2024-04-15T15:15:44.440394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588032+0000","last_change":"2024-04-15T15:15:37.561778+0000","last_active":"2024-04-15T15:15:43.588032+0000","last_peered":"2024-04-15T15:15:43.588032+0000","last_clean":"2024-04-15T15:15:43.588032+0000","last_became_active":"2024-04-15T15:15:37.561489+0000","last_became_peered":"2024-04-15T15:15:37.561489+0000","last_unstale":"2024-04-15T15:15:43.588032+0000","last_undegraded":"2024-04-15T15:15:43.588032+0000","last_fullsized":"2024-04-15T15:15:43.588032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439557+0000","last_change":"2024-04-15T15:15:38.570137+0000","last_active":"2024-04-15T15:15:44.439557+0000","last_peered":"2024-04-15T15:15:44.439557+0000","last_clean":"2024-04-15T15:15:44.439557+0000","last_became_active":"2024-04-15T15:15:38.569657+0000","last_became_peered":"2024-04-15T15:15:38.569657+0000","last_unstale":"2024-04-15T15:15:44.439557+0000","last_undegraded":"2024-04-15T15:15:44.439557+0000","last_fullsized":"2024-04-15T15:15:44.439557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769366+0000","last_change":"2024-04-15T15:15:42.584680+0000","last_active":"2024-04-15T15:15:43.769366+0000","last_peered":"2024-04-15T15:15:43.769366+0000","last_clean":"2024-04-15T15:15:43.769366+0000","last_became_active":"2024-04-15T15:15:42.584312+0000","last_became_peered":"2024-04-15T15:15:42.584312+0000","last_unstale":"2024-04-15T15:15:43.769366+0000","last_undegraded":"2024-04-15T15:15:43.769366+0000","last_fullsized":"2024-04-15T15:15:43.769366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:39.473475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440046+0000","last_change":"2024-04-15T15:15:37.565397+0000","last_active":"2024-04-15T15:15:44.440046+0000","last_peered":"2024-04-15T15:15:44.440046+0000","last_clean":"2024-04-15T15:15:44.440046+0000","last_became_active":"2024-04-15T15:15:37.565018+0000","last_became_peered":"2024-04-15T15:15:37.565018+0000","last_unstale":"2024-04-15T15:15:44.440046+0000","last_undegraded":"2024-04-15T15:15:44.440046+0000","last_fullsized":"2024-04-15T15:15:44.440046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587469+0000","last_change":"2024-04-15T15:15:38.558591+0000","last_active":"2024-04-15T15:15:43.587469+0000","last_peered":"2024-04-15T15:15:43.587469+0000","last_clean":"2024-04-15T15:15:43.587469+0000","last_became_active":"2024-04-15T15:15:38.558044+0000","last_became_peered":"2024-04-15T15:15:38.558044+0000","last_unstale":"2024-04-15T15:15:43.587469+0000","last_undegraded":"2024-04-15T15:15:43.587469+0000","last_fullsized":"2024-04-15T15:15:43.587469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:56.009007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439916+0000","last_change":"2024-04-15T15:15:42.587509+0000","last_active":"2024-04-15T15:15:44.439916+0000","last_peered":"2024-04-15T15:15:44.439916+0000","last_clean":"2024-04-15T15:15:44.439916+0000","last_became_active":"2024-04-15T15:15:42.587097+0000","last_became_peered":"2024-04-15T15:15:42.587097+0000","last_unstale":"2024-04-15T15:15:44.439916+0000","last_undegraded":"2024-04-15T15:15:44.439916+0000","last_fullsized":"2024-04-15T15:15:44.439916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:55:01.099477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588741+0000","last_change":"2024-04-15T15:15:38.566026+0000","last_active":"2024-04-15T15:15:43.588741+0000","last_peered":"2024-04-15T15:15:43.588741+0000","last_clean":"2024-04-15T15:15:43.588741+0000","last_became_active":"2024-04-15T15:15:38.565726+0000","last_became_peered":"2024-04-15T15:15:38.565726+0000","last_unstale":"2024-04-15T15:15:43.588741+0000","last_undegraded":"2024-04-15T15:15:43.588741+0000","last_fullsized":"2024-04-15T15:15:43.588741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769196+0000","last_change":"2024-04-15T15:15:38.567487+0000","last_active":"2024-04-15T15:15:43.769196+0000","last_peered":"2024-04-15T15:15:43.769196+0000","last_clean":"2024-04-15T15:15:43.769196+0000","last_became_active":"2024-04-15T15:15:38.567201+0000","last_became_peered":"2024-04-15T15:15:38.567201+0000","last_unstale":"2024-04-15T15:15:43.769196+0000","last_undegraded":"2024-04-15T15:15:43.769196+0000","last_fullsized":"2024-04-15T15:15:43.769196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440239+0000","last_change":"2024-04-15T15:15:38.569794+0000","last_active":"2024-04-15T15:15:44.440239+0000","last_peered":"2024-04-15T15:15:44.440239+0000","last_clean":"2024-04-15T15:15:44.440239+0000","last_became_active":"2024-04-15T15:15:38.569510+0000","last_became_peered":"2024-04-15T15:15:38.569510+0000","last_unstale":"2024-04-15T15:15:44.440239+0000","last_undegraded":"2024-04-15T15:15:44.440239+0000","last_fullsized":"2024-04-15T15:15:44.440239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439457+0000","last_change":"2024-04-15T15:15:38.563215+0000","last_active":"2024-04-15T15:15:44.439457+0000","last_peered":"2024-04-15T15:15:44.439457+0000","last_clean":"2024-04-15T15:15:44.439457+0000","last_became_active":"2024-04-15T15:15:38.562942+0000","last_became_peered":"2024-04-15T15:15:38.562942+0000","last_unstale":"2024-04-15T15:15:44.439457+0000","last_undegraded":"2024-04-15T15:15:44.439457+0000","last_fullsized":"2024-04-15T15:15:44.439457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440355+0000","last_change":"2024-04-15T15:15:38.567774+0000","last_active":"2024-04-15T15:15:44.440355+0000","last_peered":"2024-04-15T15:15:44.440355+0000","last_clean":"2024-04-15T15:15:44.440355+0000","last_became_active":"2024-04-15T15:15:38.567471+0000","last_became_peered":"2024-04-15T15:15:38.567471+0000","last_unstale":"2024-04-15T15:15:44.440355+0000","last_undegraded":"2024-04-15T15:15:44.440355+0000","last_fullsized":"2024-04-15T15:15:44.440355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587284+0000","last_change":"2024-04-15T15:15:38.562963+0000","last_active":"2024-04-15T15:15:43.587284+0000","last_peered":"2024-04-15T15:15:43.587284+0000","last_clean":"2024-04-15T15:15:43.587284+0000","last_became_active":"2024-04-15T15:15:38.562611+0000","last_became_peered":"2024-04-15T15:15:38.562611+0000","last_unstale":"2024-04-15T15:15:43.587284+0000","last_undegraded":"2024-04-15T15:15:43.587284+0000","last_fullsized":"2024-04-15T15:15:43.587284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588859+0000","last_change":"2024-04-15T15:15:38.566158+0000","last_active":"2024-04-15T15:15:43.588859+0000","last_peered":"2024-04-15T15:15:43.588859+0000","last_clean":"2024-04-15T15:15:43.588859+0000","last_became_active":"2024-04-15T15:15:38.565859+0000","last_became_peered":"2024-04-15T15:15:38.565859+0000","last_unstale":"2024-04-15T15:15:43.588859+0000","last_undegraded":"2024-04-15T15:15:43.588859+0000","last_fullsized":"2024-04-15T15:15:43.588859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440318+0000","last_change":"2024-04-15T15:15:38.566715+0000","last_active":"2024-04-15T15:15:44.440318+0000","last_peered":"2024-04-15T15:15:44.440318+0000","last_clean":"2024-04-15T15:15:44.440318+0000","last_became_active":"2024-04-15T15:15:38.565677+0000","last_became_peered":"2024-04-15T15:15:38.565677+0000","last_unstale":"2024-04-15T15:15:44.440318+0000","last_undegraded":"2024-04-15T15:15:44.440318+0000","last_fullsized":"2024-04-15T15:15:44.440318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439302+0000","last_change":"2024-04-15T15:15:38.565562+0000","last_active":"2024-04-15T15:15:44.439302+0000","last_peered":"2024-04-15T15:15:44.439302+0000","last_clean":"2024-04-15T15:15:44.439302+0000","last_became_active":"2024-04-15T15:15:38.564779+0000","last_became_peered":"2024-04-15T15:15:38.564779+0000","last_unstale":"2024-04-15T15:15:44.439302+0000","last_undegraded":"2024-04-15T15:15:44.439302+0000","last_fullsized":"2024-04-15T15:15:44.439302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440302+0000","last_change":"2024-04-15T15:15:38.557876+0000","last_active":"2024-04-15T15:15:44.440302+0000","last_peered":"2024-04-15T15:15:44.440302+0000","last_clean":"2024-04-15T15:15:44.440302+0000","last_became_active":"2024-04-15T15:15:38.557568+0000","last_became_peered":"2024-04-15T15:15:38.557568+0000","last_unstale":"2024-04-15T15:15:44.440302+0000","last_undegraded":"2024-04-15T15:15:44.440302+0000","last_fullsized":"2024-04-15T15:15:44.440302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440139+0000","last_change":"2024-04-15T15:15:38.568437+0000","last_active":"2024-04-15T15:15:44.440139+0000","last_peered":"2024-04-15T15:15:44.440139+0000","last_clean":"2024-04-15T15:15:44.440139+0000","last_became_active":"2024-04-15T15:15:38.567071+0000","last_became_peered":"2024-04-15T15:15:38.567071+0000","last_unstale":"2024-04-15T15:15:44.440139+0000","last_undegraded":"2024-04-15T15:15:44.440139+0000","last_fullsized":"2024-04-15T15:15:44.440139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439337+0000","last_change":"2024-04-15T15:15:38.572158+0000","last_active":"2024-04-15T15:15:44.439337+0000","last_peered":"2024-04-15T15:15:44.439337+0000","last_clean":"2024-04-15T15:15:44.439337+0000","last_became_active":"2024-04-15T15:15:38.571868+0000","last_became_peered":"2024-04-15T15:15:38.571868+0000","last_unstale":"2024-04-15T15:15:44.439337+0000","last_undegraded":"2024-04-15T15:15:44.439337+0000","last_fullsized":"2024-04-15T15:15:44.439337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588528+0000","last_change":"2024-04-15T15:15:38.556771+0000","last_active":"2024-04-15T15:15:43.588528+0000","last_peered":"2024-04-15T15:15:43.588528+0000","last_clean":"2024-04-15T15:15:43.588528+0000","last_became_active":"2024-04-15T15:15:38.556456+0000","last_became_peered":"2024-04-15T15:15:38.556456+0000","last_unstale":"2024-04-15T15:15:43.588528+0000","last_undegraded":"2024-04-15T15:15:43.588528+0000","last_fullsized":"2024-04-15T15:15:43.588528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439982+0000","last_change":"2024-04-15T15:15:38.565285+0000","last_active":"2024-04-15T15:15:44.439982+0000","last_peered":"2024-04-15T15:15:44.439982+0000","last_clean":"2024-04-15T15:15:44.439982+0000","last_became_active":"2024-04-15T15:15:38.564505+0000","last_became_peered":"2024-04-15T15:15:38.564505+0000","last_unstale":"2024-04-15T15:15:44.439982+0000","last_undegraded":"2024-04-15T15:15:44.439982+0000","last_fullsized":"2024-04-15T15:15:44.439982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:44.185650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769488+0000","last_change":"2024-04-15T15:15:38.562279+0000","last_active":"2024-04-15T15:15:43.769488+0000","last_peered":"2024-04-15T15:15:43.769488+0000","last_clean":"2024-04-15T15:15:43.769488+0000","last_became_active":"2024-04-15T15:15:38.561971+0000","last_became_peered":"2024-04-15T15:15:38.561971+0000","last_unstale":"2024-04-15T15:15:43.769488+0000","last_undegraded":"2024-04-15T15:15:43.769488+0000","last_fullsized":"2024-04-15T15:15:43.769488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440908+0000","last_change":"2024-04-15T15:15:38.569181+0000","last_active":"2024-04-15T15:15:44.440908+0000","last_peered":"2024-04-15T15:15:44.440908+0000","last_clean":"2024-04-15T15:15:44.440908+0000","last_became_active":"2024-04-15T15:15:38.566602+0000","last_became_peered":"2024-04-15T15:15:38.566602+0000","last_unstale":"2024-04-15T15:15:44.440908+0000","last_undegraded":"2024-04-15T15:15:44.440908+0000","last_fullsized":"2024-04-15T15:15:44.440908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439191+0000","last_change":"2024-04-15T15:15:38.568218+0000","last_active":"2024-04-15T15:15:44.439191+0000","last_peered":"2024-04-15T15:15:44.439191+0000","last_clean":"2024-04-15T15:15:44.439191+0000","last_became_active":"2024-04-15T15:15:38.566966+0000","last_became_peered":"2024-04-15T15:15:38.566966+0000","last_unstale":"2024-04-15T15:15:44.439191+0000","last_undegraded":"2024-04-15T15:15:44.439191+0000","last_fullsized":"2024-04-15T15:15:44.439191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586673+0000","last_change":"2024-04-15T15:15:38.563656+0000","last_active":"2024-04-15T15:15:43.586673+0000","last_peered":"2024-04-15T15:15:43.586673+0000","last_clean":"2024-04-15T15:15:43.586673+0000","last_became_active":"2024-04-15T15:15:38.563380+0000","last_became_peered":"2024-04-15T15:15:38.563380+0000","last_unstale":"2024-04-15T15:15:43.586673+0000","last_undegraded":"2024-04-15T15:15:43.586673+0000","last_fullsized":"2024-04-15T15:15:43.586673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439190+0000","last_change":"2024-04-15T15:15:38.558804+0000","last_active":"2024-04-15T15:15:44.439190+0000","last_peered":"2024-04-15T15:15:44.439190+0000","last_clean":"2024-04-15T15:15:44.439190+0000","last_became_active":"2024-04-15T15:15:38.558374+0000","last_became_peered":"2024-04-15T15:15:38.558374+0000","last_unstale":"2024-04-15T15:15:44.439190+0000","last_undegraded":"2024-04-15T15:15:44.439190+0000","last_fullsized":"2024-04-15T15:15:44.439190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:35.448266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439861+0000","last_change":"2024-04-15T15:15:38.564455+0000","last_active":"2024-04-15T15:15:44.439861+0000","last_peered":"2024-04-15T15:15:44.439861+0000","last_clean":"2024-04-15T15:15:44.439861+0000","last_became_active":"2024-04-15T15:15:38.563328+0000","last_became_peered":"2024-04-15T15:15:38.563328+0000","last_unstale":"2024-04-15T15:15:44.439861+0000","last_undegraded":"2024-04-15T15:15:44.439861+0000","last_fullsized":"2024-04-15T15:15:44.439861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588424+0000","last_change":"2024-04-15T15:15:38.563202+0000","last_active":"2024-04-15T15:15:43.588424+0000","last_peered":"2024-04-15T15:15:43.588424+0000","last_clean":"2024-04-15T15:15:43.588424+0000","last_became_active":"2024-04-15T15:15:38.562862+0000","last_became_peered":"2024-04-15T15:15:38.562862+0000","last_unstale":"2024-04-15T15:15:43.588424+0000","last_undegraded":"2024-04-15T15:15:43.588424+0000","last_fullsized":"2024-04-15T15:15:43.588424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588478+0000","last_change":"2024-04-15T15:15:38.570615+0000","last_active":"2024-04-15T15:15:43.588478+0000","last_peered":"2024-04-15T15:15:43.588478+0000","last_clean":"2024-04-15T15:15:43.588478+0000","last_became_active":"2024-04-15T15:15:38.570340+0000","last_became_peered":"2024-04-15T15:15:38.570340+0000","last_unstale":"2024-04-15T15:15:43.588478+0000","last_undegraded":"2024-04-15T15:15:43.588478+0000","last_fullsized":"2024-04-15T15:15:43.588478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588803+0000","last_change":"2024-04-15T15:15:38.567756+0000","last_active":"2024-04-15T15:15:43.588803+0000","last_peered":"2024-04-15T15:15:43.588803+0000","last_clean":"2024-04-15T15:15:43.588803+0000","last_became_active":"2024-04-15T15:15:38.567455+0000","last_became_peered":"2024-04-15T15:15:38.567455+0000","last_unstale":"2024-04-15T15:15:43.588803+0000","last_undegraded":"2024-04-15T15:15:43.588803+0000","last_fullsized":"2024-04-15T15:15:43.588803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440004+0000","last_change":"2024-04-15T15:15:38.574989+0000","last_active":"2024-04-15T15:15:44.440004+0000","last_peered":"2024-04-15T15:15:44.440004+0000","last_clean":"2024-04-15T15:15:44.440004+0000","last_became_active":"2024-04-15T15:15:38.573776+0000","last_became_peered":"2024-04-15T15:15:38.573776+0000","last_unstale":"2024-04-15T15:15:44.440004+0000","last_undegraded":"2024-04-15T15:15:44.440004+0000","last_fullsized":"2024-04-15T15:15:44.440004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:41.104693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438915+0000","last_change":"2024-04-15T15:15:38.573020+0000","last_active":"2024-04-15T15:15:44.438915+0000","last_peered":"2024-04-15T15:15:44.438915+0000","last_clean":"2024-04-15T15:15:44.438915+0000","last_became_active":"2024-04-15T15:15:38.572715+0000","last_became_peered":"2024-04-15T15:15:38.572715+0000","last_unstale":"2024-04-15T15:15:44.438915+0000","last_undegraded":"2024-04-15T15:15:44.438915+0000","last_fullsized":"2024-04-15T15:15:44.438915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439587+0000","last_change":"2024-04-15T15:15:38.568749+0000","last_active":"2024-04-15T15:15:44.439587+0000","last_peered":"2024-04-15T15:15:44.439587+0000","last_clean":"2024-04-15T15:15:44.439587+0000","last_became_active":"2024-04-15T15:15:38.568488+0000","last_became_peered":"2024-04-15T15:15:38.568488+0000","last_unstale":"2024-04-15T15:15:44.439587+0000","last_undegraded":"2024-04-15T15:15:44.439587+0000","last_fullsized":"2024-04-15T15:15:44.439587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440025+0000","last_change":"2024-04-15T15:15:38.572663+0000","last_active":"2024-04-15T15:15:44.440025+0000","last_peered":"2024-04-15T15:15:44.440025+0000","last_clean":"2024-04-15T15:15:44.440025+0000","last_became_active":"2024-04-15T15:15:38.572349+0000","last_became_peered":"2024-04-15T15:15:38.572349+0000","last_unstale":"2024-04-15T15:15:44.440025+0000","last_undegraded":"2024-04-15T15:15:44.440025+0000","last_fullsized":"2024-04-15T15:15:44.440025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439869+0000","last_change":"2024-04-15T15:15:38.567565+0000","last_active":"2024-04-15T15:15:44.439869+0000","last_peered":"2024-04-15T15:15:44.439869+0000","last_clean":"2024-04-15T15:15:44.439869+0000","last_became_active":"2024-04-15T15:15:38.567300+0000","last_became_peered":"2024-04-15T15:15:38.567300+0000","last_unstale":"2024-04-15T15:15:44.439869+0000","last_undegraded":"2024-04-15T15:15:44.439869+0000","last_fullsized":"2024-04-15T15:15:44.439869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440752+0000","last_change":"2024-04-15T15:15:38.578049+0000","last_active":"2024-04-15T15:15:44.440752+0000","last_peered":"2024-04-15T15:15:44.440752+0000","last_clean":"2024-04-15T15:15:44.440752+0000","last_became_active":"2024-04-15T15:15:38.577801+0000","last_became_peered":"2024-04-15T15:15:38.577801+0000","last_unstale":"2024-04-15T15:15:44.440752+0000","last_undegraded":"2024-04-15T15:15:44.440752+0000","last_fullsized":"2024-04-15T15:15:44.440752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439206+0000","last_change":"2024-04-15T15:15:38.561841+0000","last_active":"2024-04-15T15:15:44.439206+0000","last_peered":"2024-04-15T15:15:44.439206+0000","last_clean":"2024-04-15T15:15:44.439206+0000","last_became_active":"2024-04-15T15:15:38.561554+0000","last_became_peered":"2024-04-15T15:15:38.561554+0000","last_unstale":"2024-04-15T15:15:44.439206+0000","last_undegraded":"2024-04-15T15:15:44.439206+0000","last_fullsized":"2024-04-15T15:15:44.439206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587614+0000","last_change":"2024-04-15T15:15:38.560883+0000","last_active":"2024-04-15T15:15:43.587614+0000","last_peered":"2024-04-15T15:15:43.587614+0000","last_clean":"2024-04-15T15:15:43.587614+0000","last_became_active":"2024-04-15T15:15:38.560562+0000","last_became_peered":"2024-04-15T15:15:38.560562+0000","last_unstale":"2024-04-15T15:15:43.587614+0000","last_undegraded":"2024-04-15T15:15:43.587614+0000","last_fullsized":"2024-04-15T15:15:43.587614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439744+0000","last_change":"2024-04-15T15:15:38.565038+0000","last_active":"2024-04-15T15:15:44.439744+0000","last_peered":"2024-04-15T15:15:44.439744+0000","last_clean":"2024-04-15T15:15:44.439744+0000","last_became_active":"2024-04-15T15:15:38.564551+0000","last_became_peered":"2024-04-15T15:15:38.564551+0000","last_unstale":"2024-04-15T15:15:44.439744+0000","last_undegraded":"2024-04-15T15:15:44.439744+0000","last_fullsized":"2024-04-15T15:15:44.439744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":104213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":695037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":104087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":104247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":868352,"data_stored":695093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:05.468 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.606+0000 7fc265a14640 1 Processor -- start 2024-04-15T15:16:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.606+0000 7fc265a14640 1 -- start start 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc265a14640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015e700 0x7fc2600b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc265a14640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2600d2360 0x7fc2600d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc265a14640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc26015eae0 0x7fc2600d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc265a14640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc2600b53b0 con 0x7fc2600d2360 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc265a14640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc2600b47a0 con 0x7fc26015eae0 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc265a14640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc2600b4a80 con 0x7fc26015e700 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2600d2360 0x7fc2600d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc2600d2360 0x7fc2600d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42348/0 (socket says 172.21.15.46:42348) 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2572177211 learned_addr learned my addr 172.21.15.46:0/2572177211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25effd640 1 --2- 172.21.15.46:0/2572177211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015e700 0x7fc2600b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25f7fe640 1 --2- 172.21.15.46:0/2572177211 >> v2:172.21.15.78:3300/0 conn(0x7fc26015eae0 0x7fc2600d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2572177211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015e700 msgr2=0x7fc2600b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 --2- 172.21.15.46:0/2572177211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015e700 0x7fc2600b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2572177211 >> v2:172.21.15.78:3300/0 conn(0x7fc26015eae0 msgr2=0x7fc2600d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 --2- 172.21.15.46:0/2572177211 >> v2:172.21.15.78:3300/0 conn(0x7fc26015eae0 0x7fc2600d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.607+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2572177211 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2600b4d60 con 0x7fc2600d2360 2024-04-15T15:16:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.608+0000 7fc25e7fc640 1 --2- 172.21.15.46:0/2572177211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2600d2360 0x7fc2600d2b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc26008c780 tx=0x7fc25402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7c38c636f980f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.608+0000 7fc25dffb640 1 -- 172.21.15.46:0/2572177211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254035020 con 0x7fc2600d2360 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.608+0000 7fc25dffb640 1 -- 172.21.15.46:0/2572177211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc254031870 con 0x7fc2600d2360 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.608+0000 7fc25dffb640 1 -- 172.21.15.46:0/2572177211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254031c40 con 0x7fc2600d2360 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 -- 172.21.15.46:0/2572177211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2600d2360 msgr2=0x7fc2600d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 --2- 172.21.15.46:0/2572177211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2600d2360 0x7fc2600d2b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc26008c780 tx=0x7fc25402b910 comp rx=0 tx=0).stop 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 -- 172.21.15.46:0/2572177211 shutdown_connections 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 --2- 172.21.15.46:0/2572177211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2600d2360 0x7fc2600d2b60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 --2- 172.21.15.46:0/2572177211 >> v2:172.21.15.78:3300/0 conn(0x7fc26015eae0 0x7fc2600d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 --2- 172.21.15.46:0/2572177211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015e700 0x7fc2600b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 -- 172.21.15.46:0/2572177211 >> 172.21.15.46:0/2572177211 conn(0x7fc26008bf60 msgr2=0x7fc2600d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 -- 172.21.15.46:0/2572177211 shutdown_connections 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.609+0000 7fc265a14640 1 -- 172.21.15.46:0/2572177211 wait complete. 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.610+0000 7fc265a14640 1 Processor -- start 2024-04-15T15:16:05.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.610+0000 7fc265a14640 1 -- start start 2024-04-15T15:16:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc265a14640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2600d2360 0x7fc2600cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc265a14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 0x7fc2600c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc265a14640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015eae0 0x7fc2600c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc265a14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc2600b5d80 con 0x7fc26015e700 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc265a14640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc2600b4450 con 0x7fc2600d2360 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc265a14640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc2600d1d40 con 0x7fc26015eae0 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 0x7fc2600c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 0x7fc2600c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42350/0 (socket says 172.21.15.46:42350) 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2037357299 learned_addr learned my addr 172.21.15.46:0/2037357299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25effd640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2600d2360 0x7fc2600cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25f7fe640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015eae0 0x7fc2600c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015eae0 msgr2=0x7fc2600c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015eae0 0x7fc2600c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2600d2360 msgr2=0x7fc2600cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2600d2360 0x7fc2600cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 -- 172.21.15.46:0/2037357299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2600cd2b0 con 0x7fc26015e700 2024-04-15T15:16:05.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25f7fe640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015eae0 0x7fc2600c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.611+0000 7fc25e7fc640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 0x7fc2600c7f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc25402b950 tx=0x7fc2540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.612+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254040070 con 0x7fc26015e700 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.612+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc254031e10 con 0x7fc26015e700 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.612+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2600cd590 con 0x7fc26015e700 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.612+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254035040 con 0x7fc26015e700 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.612+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2600cdb50 con 0x7fc26015e700 2024-04-15T15:16:05.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.613+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2600689e0 con 0x7fc26015e700 2024-04-15T15:16:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.613+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc25404b060 con 0x7fc26015e700 2024-04-15T15:16:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.614+0000 7fc247fff640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc24c06c8f0 0x7fc24c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.614+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc2540adac0 con 0x7fc26015e700 2024-04-15T15:16:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.614+0000 7fc25effd640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc24c06c8f0 0x7fc24c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.614+0000 7fc25effd640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc24c06c8f0 0x7fc24c06edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc2500074e0 tx=0x7fc25000d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.618+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc25407bbb0 con 0x7fc26015e700 2024-04-15T15:16:05.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.801+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2600ba500 con 0x7fc24c06c8f0 2024-04-15T15:16:05.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.810+0000 7fc247fff640 1 -- 172.21.15.46:0/2037357299 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332533 (secure 0 0 0) 0x7fc2600ba500 con 0x7fc24c06c8f0 2024-04-15T15:16:05.815 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:05.817 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:16:05.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc24c06c8f0 msgr2=0x7fc24c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc24c06c8f0 0x7fc24c06edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc2500074e0 tx=0x7fc25000d040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 msgr2=0x7fc2600c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 0x7fc2600c7f70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc25402b950 tx=0x7fc2540046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 shutdown_connections 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc24c06c8f0 0x7fc24c06edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc26015eae0 0x7fc2600c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc26015e700 0x7fc2600c7f70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 --2- 172.21.15.46:0/2037357299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2600d2360 0x7fc2600cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 >> 172.21.15.46:0/2037357299 conn(0x7fc26008bf60 msgr2=0x7fc2600d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 shutdown_connections 2024-04-15T15:16:05.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.813+0000 7fc265a14640 1 -- 172.21.15.46:0/2037357299 wait complete. 2024-04-15T15:16:05.826 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-15T15:16:04.359458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":231928,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742792,"statfs":{"total":773094113280,"available":772856619008,"internally_reserved":0,"allocated":3350528,"data_stored":2022712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439492+0000","last_change":"2024-04-15T15:15:37.566709+0000","last_active":"2024-04-15T15:15:44.439492+0000","last_peered":"2024-04-15T15:15:44.439492+0000","last_clean":"2024-04-15T15:15:44.439492+0000","last_became_active":"2024-04-15T15:15:37.566073+0000","last_became_peered":"2024-04-15T15:15:37.566073+0000","last_unstale":"2024-04-15T15:15:44.439492+0000","last_undegraded":"2024-04-15T15:15:44.439492+0000","last_fullsized":"2024-04-15T15:15:44.439492+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:40:55.907098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440303+0000","last_change":"2024-04-15T15:15:38.565494+0000","last_active":"2024-04-15T15:15:44.440303+0000","last_peered":"2024-04-15T15:15:44.440303+0000","last_clean":"2024-04-15T15:15:44.440303+0000","last_became_active":"2024-04-15T15:15:38.563911+0000","last_became_peered":"2024-04-15T15:15:38.563911+0000","last_unstale":"2024-04-15T15:15:44.440303+0000","last_undegraded":"2024-04-15T15:15:44.440303+0000","last_fullsized":"2024-04-15T15:15:44.440303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439886+0000","last_change":"2024-04-15T15:15:42.661119+0000","last_active":"2024-04-15T15:15:44.439886+0000","last_peered":"2024-04-15T15:15:44.439886+0000","last_clean":"2024-04-15T15:15:44.439886+0000","last_became_active":"2024-04-15T15:15:42.660440+0000","last_became_peered":"2024-04-15T15:15:42.660440+0000","last_unstale":"2024-04-15T15:15:44.439886+0000","last_undegraded":"2024-04-15T15:15:44.439886+0000","last_fullsized":"2024-04-15T15:15:44.439886+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440400+0000","last_change":"2024-04-15T15:15:37.563320+0000","last_active":"2024-04-15T15:15:44.440400+0000","last_peered":"2024-04-15T15:15:44.440400+0000","last_clean":"2024-04-15T15:15:44.440400+0000","last_became_active":"2024-04-15T15:15:37.563058+0000","last_became_peered":"2024-04-15T15:15:37.563058+0000","last_unstale":"2024-04-15T15:15:44.440400+0000","last_undegraded":"2024-04-15T15:15:44.440400+0000","last_fullsized":"2024-04-15T15:15:44.440400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439174+0000","last_change":"2024-04-15T15:15:38.573022+0000","last_active":"2024-04-15T15:15:44.439174+0000","last_peered":"2024-04-15T15:15:44.439174+0000","last_clean":"2024-04-15T15:15:44.439174+0000","last_became_active":"2024-04-15T15:15:38.572726+0000","last_became_peered":"2024-04-15T15:15:38.572726+0000","last_unstale":"2024-04-15T15:15:44.439174+0000","last_undegraded":"2024-04-15T15:15:44.439174+0000","last_fullsized":"2024-04-15T15:15:44.439174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587093+0000","last_change":"2024-04-15T15:15:42.590697+0000","last_active":"2024-04-15T15:15:43.587093+0000","last_peered":"2024-04-15T15:15:43.587093+0000","last_clean":"2024-04-15T15:15:43.587093+0000","last_became_active":"2024-04-15T15:15:42.590272+0000","last_became_peered":"2024-04-15T15:15:42.590272+0000","last_unstale":"2024-04-15T15:15:43.587093+0000","last_undegraded":"2024-04-15T15:15:43.587093+0000","last_fullsized":"2024-04-15T15:15:43.587093+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:10.918812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439863+0000","last_change":"2024-04-15T15:15:37.564953+0000","last_active":"2024-04-15T15:15:44.439863+0000","last_peered":"2024-04-15T15:15:44.439863+0000","last_clean":"2024-04-15T15:15:44.439863+0000","last_became_active":"2024-04-15T15:15:37.564653+0000","last_became_peered":"2024-04-15T15:15:37.564653+0000","last_unstale":"2024-04-15T15:15:44.439863+0000","last_undegraded":"2024-04-15T15:15:44.439863+0000","last_fullsized":"2024-04-15T15:15:44.439863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440820+0000","last_change":"2024-04-15T15:15:38.568986+0000","last_active":"2024-04-15T15:15:44.440820+0000","last_peered":"2024-04-15T15:15:44.440820+0000","last_clean":"2024-04-15T15:15:44.440820+0000","last_became_active":"2024-04-15T15:15:38.566559+0000","last_became_peered":"2024-04-15T15:15:38.566559+0000","last_unstale":"2024-04-15T15:15:44.440820+0000","last_undegraded":"2024-04-15T15:15:44.440820+0000","last_fullsized":"2024-04-15T15:15:44.440820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587815+0000","last_change":"2024-04-15T15:15:42.592706+0000","last_active":"2024-04-15T15:15:43.587815+0000","last_peered":"2024-04-15T15:15:43.587815+0000","last_clean":"2024-04-15T15:15:43.587815+0000","last_became_active":"2024-04-15T15:15:42.592134+0000","last_became_peered":"2024-04-15T15:15:42.592134+0000","last_unstale":"2024-04-15T15:15:43.587815+0000","last_undegraded":"2024-04-15T15:15:43.587815+0000","last_fullsized":"2024-04-15T15:15:43.587815+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:41.894568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439525+0000","last_change":"2024-04-15T15:15:37.567857+0000","last_active":"2024-04-15T15:15:44.439525+0000","last_peered":"2024-04-15T15:15:44.439525+0000","last_clean":"2024-04-15T15:15:44.439525+0000","last_became_active":"2024-04-15T15:15:37.567560+0000","last_became_peered":"2024-04-15T15:15:37.567560+0000","last_unstale":"2024-04-15T15:15:44.439525+0000","last_undegraded":"2024-04-15T15:15:44.439525+0000","last_fullsized":"2024-04-15T15:15:44.439525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:36:21.001555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440328+0000","last_change":"2024-04-15T15:15:38.568788+0000","last_active":"2024-04-15T15:15:44.440328+0000","last_peered":"2024-04-15T15:15:44.440328+0000","last_clean":"2024-04-15T15:15:44.440328+0000","last_became_active":"2024-04-15T15:15:38.567501+0000","last_became_peered":"2024-04-15T15:15:38.567501+0000","last_unstale":"2024-04-15T15:15:44.440328+0000","last_undegraded":"2024-04-15T15:15:44.440328+0000","last_fullsized":"2024-04-15T15:15:44.440328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587164+0000","last_change":"2024-04-15T15:15:42.587928+0000","last_active":"2024-04-15T15:15:43.587164+0000","last_peered":"2024-04-15T15:15:43.587164+0000","last_clean":"2024-04-15T15:15:43.587164+0000","last_became_active":"2024-04-15T15:15:42.587518+0000","last_became_peered":"2024-04-15T15:15:42.587518+0000","last_unstale":"2024-04-15T15:15:43.587164+0000","last_undegraded":"2024-04-15T15:15:43.587164+0000","last_fullsized":"2024-04-15T15:15:43.587164+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:50.908590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587148+0000","last_change":"2024-04-15T15:15:38.560568+0000","last_active":"2024-04-15T15:15:43.587148+0000","last_peered":"2024-04-15T15:15:43.587148+0000","last_clean":"2024-04-15T15:15:43.587148+0000","last_became_active":"2024-04-15T15:15:38.560292+0000","last_became_peered":"2024-04-15T15:15:38.560292+0000","last_unstale":"2024-04-15T15:15:43.587148+0000","last_undegraded":"2024-04-15T15:15:43.587148+0000","last_fullsized":"2024-04-15T15:15:43.587148+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440210+0000","last_change":"2024-04-15T15:15:37.570166+0000","last_active":"2024-04-15T15:15:44.440210+0000","last_peered":"2024-04-15T15:15:44.440210+0000","last_clean":"2024-04-15T15:15:44.440210+0000","last_became_active":"2024-04-15T15:15:37.569862+0000","last_became_peered":"2024-04-15T15:15:37.569862+0000","last_unstale":"2024-04-15T15:15:44.440210+0000","last_undegraded":"2024-04-15T15:15:44.440210+0000","last_fullsized":"2024-04-15T15:15:44.440210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:55:13.182192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588794+0000","last_change":"2024-04-15T15:15:42.590929+0000","last_active":"2024-04-15T15:15:43.588794+0000","last_peered":"2024-04-15T15:15:43.588794+0000","last_clean":"2024-04-15T15:15:43.588794+0000","last_became_active":"2024-04-15T15:15:42.589759+0000","last_became_peered":"2024-04-15T15:15:42.589759+0000","last_unstale":"2024-04-15T15:15:43.588794+0000","last_undegraded":"2024-04-15T15:15:43.588794+0000","last_fullsized":"2024-04-15T15:15:43.588794+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439955+0000","last_change":"2024-04-15T15:15:38.563606+0000","last_active":"2024-04-15T15:15:44.439955+0000","last_peered":"2024-04-15T15:15:44.439955+0000","last_clean":"2024-04-15T15:15:44.439955+0000","last_became_active":"2024-04-15T15:15:38.562885+0000","last_became_peered":"2024-04-15T15:15:38.562885+0000","last_unstale":"2024-04-15T15:15:44.439955+0000","last_undegraded":"2024-04-15T15:15:44.439955+0000","last_fullsized":"2024-04-15T15:15:44.439955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587327+0000","last_change":"2024-04-15T15:15:37.562063+0000","last_active":"2024-04-15T15:15:43.587327+0000","last_peered":"2024-04-15T15:15:43.587327+0000","last_clean":"2024-04-15T15:15:43.587327+0000","last_became_active":"2024-04-15T15:15:37.561757+0000","last_became_peered":"2024-04-15T15:15:37.561757+0000","last_unstale":"2024-04-15T15:15:43.587327+0000","last_undegraded":"2024-04-15T15:15:43.587327+0000","last_fullsized":"2024-04-15T15:15:43.587327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588354+0000","last_change":"2024-04-15T15:15:42.591439+0000","last_active":"2024-04-15T15:15:43.588354+0000","last_peered":"2024-04-15T15:15:43.588354+0000","last_clean":"2024-04-15T15:15:43.588354+0000","last_became_active":"2024-04-15T15:15:42.591030+0000","last_became_peered":"2024-04-15T15:15:42.591030+0000","last_unstale":"2024-04-15T15:15:43.588354+0000","last_undegraded":"2024-04-15T15:15:43.588354+0000","last_fullsized":"2024-04-15T15:15:43.588354+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:19.368841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439666+0000","last_change":"2024-04-15T15:15:38.566770+0000","last_active":"2024-04-15T15:15:44.439666+0000","last_peered":"2024-04-15T15:15:44.439666+0000","last_clean":"2024-04-15T15:15:44.439666+0000","last_became_active":"2024-04-15T15:15:38.566484+0000","last_became_peered":"2024-04-15T15:15:38.566484+0000","last_unstale":"2024-04-15T15:15:44.439666+0000","last_undegraded":"2024-04-15T15:15:44.439666+0000","last_fullsized":"2024-04-15T15:15:44.439666+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586733+0000","last_change":"2024-04-15T15:15:37.559565+0000","last_active":"2024-04-15T15:15:43.586733+0000","last_peered":"2024-04-15T15:15:43.586733+0000","last_clean":"2024-04-15T15:15:43.586733+0000","last_became_active":"2024-04-15T15:15:37.558121+0000","last_became_peered":"2024-04-15T15:15:37.558121+0000","last_unstale":"2024-04-15T15:15:43.586733+0000","last_undegraded":"2024-04-15T15:15:43.586733+0000","last_fullsized":"2024-04-15T15:15:43.586733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.655888+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.655425+0000","last_became_peered":"2024-04-15T15:15:42.655425+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587565+0000","last_change":"2024-04-15T15:15:38.558135+0000","last_active":"2024-04-15T15:15:43.587565+0000","last_peered":"2024-04-15T15:15:43.587565+0000","last_clean":"2024-04-15T15:15:43.587565+0000","last_became_active":"2024-04-15T15:15:38.557849+0000","last_became_peered":"2024-04-15T15:15:38.557849+0000","last_unstale":"2024-04-15T15:15:43.587565+0000","last_undegraded":"2024-04-15T15:15:43.587565+0000","last_fullsized":"2024-04-15T15:15:43.587565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:08.649907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439485+0000","last_change":"2024-04-15T15:15:37.568252+0000","last_active":"2024-04-15T15:15:44.439485+0000","last_peered":"2024-04-15T15:15:44.439485+0000","last_clean":"2024-04-15T15:15:44.439485+0000","last_became_active":"2024-04-15T15:15:37.567982+0000","last_became_peered":"2024-04-15T15:15:37.567982+0000","last_unstale":"2024-04-15T15:15:44.439485+0000","last_undegraded":"2024-04-15T15:15:44.439485+0000","last_fullsized":"2024-04-15T15:15:44.439485+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588452+0000","last_change":"2024-04-15T15:15:42.593489+0000","last_active":"2024-04-15T15:15:43.588452+0000","last_peered":"2024-04-15T15:15:43.588452+0000","last_clean":"2024-04-15T15:15:43.588452+0000","last_became_active":"2024-04-15T15:15:42.593152+0000","last_became_peered":"2024-04-15T15:15:42.593152+0000","last_unstale":"2024-04-15T15:15:43.588452+0000","last_undegraded":"2024-04-15T15:15:43.588452+0000","last_fullsized":"2024-04-15T15:15:43.588452+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439610+0000","last_change":"2024-04-15T15:15:38.559585+0000","last_active":"2024-04-15T15:15:44.439610+0000","last_peered":"2024-04-15T15:15:44.439610+0000","last_clean":"2024-04-15T15:15:44.439610+0000","last_became_active":"2024-04-15T15:15:38.559225+0000","last_became_peered":"2024-04-15T15:15:38.559225+0000","last_unstale":"2024-04-15T15:15:44.439610+0000","last_undegraded":"2024-04-15T15:15:44.439610+0000","last_fullsized":"2024-04-15T15:15:44.439610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.589097+0000","last_change":"2024-04-15T15:15:37.558650+0000","last_active":"2024-04-15T15:15:43.589097+0000","last_peered":"2024-04-15T15:15:43.589097+0000","last_clean":"2024-04-15T15:15:43.589097+0000","last_became_active":"2024-04-15T15:15:37.558128+0000","last_became_peered":"2024-04-15T15:15:37.558128+0000","last_unstale":"2024-04-15T15:15:43.589097+0000","last_undegraded":"2024-04-15T15:15:43.589097+0000","last_fullsized":"2024-04-15T15:15:43.589097+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587388+0000","last_change":"2024-04-15T15:15:42.655806+0000","last_active":"2024-04-15T15:15:43.587388+0000","last_peered":"2024-04-15T15:15:43.587388+0000","last_clean":"2024-04-15T15:15:43.587388+0000","last_became_active":"2024-04-15T15:15:42.655431+0000","last_became_peered":"2024-04-15T15:15:42.655431+0000","last_unstale":"2024-04-15T15:15:43.587388+0000","last_undegraded":"2024-04-15T15:15:43.587388+0000","last_fullsized":"2024-04-15T15:15:43.587388+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:10.032080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440420+0000","last_change":"2024-04-15T15:15:38.567366+0000","last_active":"2024-04-15T15:15:44.440420+0000","last_peered":"2024-04-15T15:15:44.440420+0000","last_clean":"2024-04-15T15:15:44.440420+0000","last_became_active":"2024-04-15T15:15:38.567064+0000","last_became_peered":"2024-04-15T15:15:38.567064+0000","last_unstale":"2024-04-15T15:15:44.440420+0000","last_undegraded":"2024-04-15T15:15:44.440420+0000","last_fullsized":"2024-04-15T15:15:44.440420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587807+0000","last_change":"2024-04-15T15:15:37.554351+0000","last_active":"2024-04-15T15:15:43.587807+0000","last_peered":"2024-04-15T15:15:43.587807+0000","last_clean":"2024-04-15T15:15:43.587807+0000","last_became_active":"2024-04-15T15:15:37.554062+0000","last_became_peered":"2024-04-15T15:15:37.554062+0000","last_unstale":"2024-04-15T15:15:43.587807+0000","last_undegraded":"2024-04-15T15:15:43.587807+0000","last_fullsized":"2024-04-15T15:15:43.587807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438909+0000","last_change":"2024-04-15T15:15:42.659938+0000","last_active":"2024-04-15T15:15:44.438909+0000","last_peered":"2024-04-15T15:15:44.438909+0000","last_clean":"2024-04-15T15:15:44.438909+0000","last_became_active":"2024-04-15T15:15:42.659252+0000","last_became_peered":"2024-04-15T15:15:42.659252+0000","last_unstale":"2024-04-15T15:15:44.438909+0000","last_undegraded":"2024-04-15T15:15:44.438909+0000","last_fullsized":"2024-04-15T15:15:44.438909+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:48.210859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587105+0000","last_change":"2024-04-15T15:15:38.561075+0000","last_active":"2024-04-15T15:15:43.587105+0000","last_peered":"2024-04-15T15:15:43.587105+0000","last_clean":"2024-04-15T15:15:43.587105+0000","last_became_active":"2024-04-15T15:15:38.559163+0000","last_became_peered":"2024-04-15T15:15:38.559163+0000","last_unstale":"2024-04-15T15:15:43.587105+0000","last_undegraded":"2024-04-15T15:15:43.587105+0000","last_fullsized":"2024-04-15T15:15:43.587105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440117+0000","last_change":"2024-04-15T15:15:37.568261+0000","last_active":"2024-04-15T15:15:44.440117+0000","last_peered":"2024-04-15T15:15:44.440117+0000","last_clean":"2024-04-15T15:15:44.440117+0000","last_became_active":"2024-04-15T15:15:37.567997+0000","last_became_peered":"2024-04-15T15:15:37.567997+0000","last_unstale":"2024-04-15T15:15:44.440117+0000","last_undegraded":"2024-04-15T15:15:44.440117+0000","last_fullsized":"2024-04-15T15:15:44.440117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:47:24.543446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:42.658160+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:42.657766+0000","last_became_peered":"2024-04-15T15:15:42.657766+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440434+0000","last_change":"2024-04-15T15:15:38.563836+0000","last_active":"2024-04-15T15:15:44.440434+0000","last_peered":"2024-04-15T15:15:44.440434+0000","last_clean":"2024-04-15T15:15:44.440434+0000","last_became_active":"2024-04-15T15:15:38.563487+0000","last_became_peered":"2024-04-15T15:15:38.563487+0000","last_unstale":"2024-04-15T15:15:44.440434+0000","last_undegraded":"2024-04-15T15:15:44.440434+0000","last_fullsized":"2024-04-15T15:15:44.440434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587767+0000","last_change":"2024-04-15T15:15:37.563786+0000","last_active":"2024-04-15T15:15:43.587767+0000","last_peered":"2024-04-15T15:15:43.587767+0000","last_clean":"2024-04-15T15:15:43.587767+0000","last_became_active":"2024-04-15T15:15:37.563491+0000","last_became_peered":"2024-04-15T15:15:37.563491+0000","last_unstale":"2024-04-15T15:15:43.587767+0000","last_undegraded":"2024-04-15T15:15:43.587767+0000","last_fullsized":"2024-04-15T15:15:43.587767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586975+0000","last_change":"2024-04-15T15:15:42.593439+0000","last_active":"2024-04-15T15:15:43.586975+0000","last_peered":"2024-04-15T15:15:43.586975+0000","last_clean":"2024-04-15T15:15:43.586975+0000","last_became_active":"2024-04-15T15:15:42.591555+0000","last_became_peered":"2024-04-15T15:15:42.591555+0000","last_unstale":"2024-04-15T15:15:43.586975+0000","last_undegraded":"2024-04-15T15:15:43.586975+0000","last_fullsized":"2024-04-15T15:15:43.586975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588422+0000","last_change":"2024-04-15T15:15:38.560053+0000","last_active":"2024-04-15T15:15:43.588422+0000","last_peered":"2024-04-15T15:15:43.588422+0000","last_clean":"2024-04-15T15:15:43.588422+0000","last_became_active":"2024-04-15T15:15:38.559462+0000","last_became_peered":"2024-04-15T15:15:38.559462+0000","last_unstale":"2024-04-15T15:15:43.588422+0000","last_undegraded":"2024-04-15T15:15:43.588422+0000","last_fullsized":"2024-04-15T15:15:43.588422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:50.427117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440424+0000","last_change":"2024-04-15T15:15:37.561463+0000","last_active":"2024-04-15T15:15:44.440424+0000","last_peered":"2024-04-15T15:15:44.440424+0000","last_clean":"2024-04-15T15:15:44.440424+0000","last_became_active":"2024-04-15T15:15:37.560849+0000","last_became_peered":"2024-04-15T15:15:37.560849+0000","last_unstale":"2024-04-15T15:15:44.440424+0000","last_undegraded":"2024-04-15T15:15:44.440424+0000","last_fullsized":"2024-04-15T15:15:44.440424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587727+0000","last_change":"2024-04-15T15:15:42.594805+0000","last_active":"2024-04-15T15:15:43.587727+0000","last_peered":"2024-04-15T15:15:43.587727+0000","last_clean":"2024-04-15T15:15:43.587727+0000","last_became_active":"2024-04-15T15:15:42.594322+0000","last_became_peered":"2024-04-15T15:15:42.594322+0000","last_unstale":"2024-04-15T15:15:43.587727+0000","last_undegraded":"2024-04-15T15:15:43.587727+0000","last_fullsized":"2024-04-15T15:15:43.587727+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:02.798272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586925+0000","last_change":"2024-04-15T15:15:38.559721+0000","last_active":"2024-04-15T15:15:43.586925+0000","last_peered":"2024-04-15T15:15:43.586925+0000","last_clean":"2024-04-15T15:15:43.586925+0000","last_became_active":"2024-04-15T15:15:38.559440+0000","last_became_peered":"2024-04-15T15:15:38.559440+0000","last_unstale":"2024-04-15T15:15:43.586925+0000","last_undegraded":"2024-04-15T15:15:43.586925+0000","last_fullsized":"2024-04-15T15:15:43.586925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769205+0000","last_change":"2024-04-15T15:15:37.562651+0000","last_active":"2024-04-15T15:15:43.769205+0000","last_peered":"2024-04-15T15:15:43.769205+0000","last_clean":"2024-04-15T15:15:43.769205+0000","last_became_active":"2024-04-15T15:15:37.562267+0000","last_became_peered":"2024-04-15T15:15:37.562267+0000","last_unstale":"2024-04-15T15:15:43.769205+0000","last_undegraded":"2024-04-15T15:15:43.769205+0000","last_fullsized":"2024-04-15T15:15:43.769205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587782+0000","last_change":"2024-04-15T15:15:42.595294+0000","last_active":"2024-04-15T15:15:43.587782+0000","last_peered":"2024-04-15T15:15:43.587782+0000","last_clean":"2024-04-15T15:15:43.587782+0000","last_became_active":"2024-04-15T15:15:42.594395+0000","last_became_peered":"2024-04-15T15:15:42.594395+0000","last_unstale":"2024-04-15T15:15:43.587782+0000","last_undegraded":"2024-04-15T15:15:43.587782+0000","last_fullsized":"2024-04-15T15:15:43.587782+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438893+0000","last_change":"2024-04-15T15:15:38.566568+0000","last_active":"2024-04-15T15:15:44.438893+0000","last_peered":"2024-04-15T15:15:44.438893+0000","last_clean":"2024-04-15T15:15:44.438893+0000","last_became_active":"2024-04-15T15:15:38.566261+0000","last_became_peered":"2024-04-15T15:15:38.566261+0000","last_unstale":"2024-04-15T15:15:44.438893+0000","last_undegraded":"2024-04-15T15:15:44.438893+0000","last_fullsized":"2024-04-15T15:15:44.438893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439388+0000","last_change":"2024-04-15T15:15:37.567337+0000","last_active":"2024-04-15T15:15:44.439388+0000","last_peered":"2024-04-15T15:15:44.439388+0000","last_clean":"2024-04-15T15:15:44.439388+0000","last_became_active":"2024-04-15T15:15:37.567042+0000","last_became_peered":"2024-04-15T15:15:37.567042+0000","last_unstale":"2024-04-15T15:15:44.439388+0000","last_undegraded":"2024-04-15T15:15:44.439388+0000","last_fullsized":"2024-04-15T15:15:44.439388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439562+0000","last_change":"2024-04-15T15:15:42.657091+0000","last_active":"2024-04-15T15:15:44.439562+0000","last_peered":"2024-04-15T15:15:44.439562+0000","last_clean":"2024-04-15T15:15:44.439562+0000","last_became_active":"2024-04-15T15:15:42.656638+0000","last_became_peered":"2024-04-15T15:15:42.656638+0000","last_unstale":"2024-04-15T15:15:44.439562+0000","last_undegraded":"2024-04-15T15:15:44.439562+0000","last_fullsized":"2024-04-15T15:15:44.439562+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:46.336878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439991+0000","last_change":"2024-04-15T15:15:38.576736+0000","last_active":"2024-04-15T15:15:44.439991+0000","last_peered":"2024-04-15T15:15:44.439991+0000","last_clean":"2024-04-15T15:15:44.439991+0000","last_became_active":"2024-04-15T15:15:38.574326+0000","last_became_peered":"2024-04-15T15:15:38.574326+0000","last_unstale":"2024-04-15T15:15:44.439991+0000","last_undegraded":"2024-04-15T15:15:44.439991+0000","last_fullsized":"2024-04-15T15:15:44.439991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440769+0000","last_change":"2024-04-15T15:15:37.557351+0000","last_active":"2024-04-15T15:15:44.440769+0000","last_peered":"2024-04-15T15:15:44.440769+0000","last_clean":"2024-04-15T15:15:44.440769+0000","last_became_active":"2024-04-15T15:15:37.557036+0000","last_became_peered":"2024-04-15T15:15:37.557036+0000","last_unstale":"2024-04-15T15:15:44.440769+0000","last_undegraded":"2024-04-15T15:15:44.440769+0000","last_fullsized":"2024-04-15T15:15:44.440769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":177,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439564+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:15:44.439564+0000","last_peered":"2024-04-15T15:15:44.439564+0000","last_clean":"2024-04-15T15:15:44.439564+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:15:44.439564+0000","last_undegraded":"2024-04-15T15:15:44.439564+0000","last_fullsized":"2024-04-15T15:15:44.439564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588337+0000","last_change":"2024-04-15T15:15:42.658428+0000","last_active":"2024-04-15T15:15:43.588337+0000","last_peered":"2024-04-15T15:15:43.588337+0000","last_clean":"2024-04-15T15:15:43.588337+0000","last_became_active":"2024-04-15T15:15:42.658014+0000","last_became_peered":"2024-04-15T15:15:42.658014+0000","last_unstale":"2024-04-15T15:15:43.588337+0000","last_undegraded":"2024-04-15T15:15:43.588337+0000","last_fullsized":"2024-04-15T15:15:43.588337+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439749+0000","last_change":"2024-04-15T15:15:38.572859+0000","last_active":"2024-04-15T15:15:44.439749+0000","last_peered":"2024-04-15T15:15:44.439749+0000","last_clean":"2024-04-15T15:15:44.439749+0000","last_became_active":"2024-04-15T15:15:38.572580+0000","last_became_peered":"2024-04-15T15:15:38.572580+0000","last_unstale":"2024-04-15T15:15:44.439749+0000","last_undegraded":"2024-04-15T15:15:44.439749+0000","last_fullsized":"2024-04-15T15:15:44.439749+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769429+0000","last_change":"2024-04-15T15:15:37.555795+0000","last_active":"2024-04-15T15:15:43.769429+0000","last_peered":"2024-04-15T15:15:43.769429+0000","last_clean":"2024-04-15T15:15:43.769429+0000","last_became_active":"2024-04-15T15:15:37.555502+0000","last_became_peered":"2024-04-15T15:15:37.555502+0000","last_unstale":"2024-04-15T15:15:43.769429+0000","last_undegraded":"2024-04-15T15:15:43.769429+0000","last_fullsized":"2024-04-15T15:15:43.769429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587412+0000","last_change":"2024-04-15T15:15:42.655842+0000","last_active":"2024-04-15T15:15:43.587412+0000","last_peered":"2024-04-15T15:15:43.587412+0000","last_clean":"2024-04-15T15:15:43.587412+0000","last_became_active":"2024-04-15T15:15:42.655470+0000","last_became_peered":"2024-04-15T15:15:42.655470+0000","last_unstale":"2024-04-15T15:15:43.587412+0000","last_undegraded":"2024-04-15T15:15:43.587412+0000","last_fullsized":"2024-04-15T15:15:43.587412+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:42.152666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440874+0000","last_change":"2024-04-15T15:15:38.573853+0000","last_active":"2024-04-15T15:15:44.440874+0000","last_peered":"2024-04-15T15:15:44.440874+0000","last_clean":"2024-04-15T15:15:44.440874+0000","last_became_active":"2024-04-15T15:15:38.573585+0000","last_became_peered":"2024-04-15T15:15:38.573585+0000","last_unstale":"2024-04-15T15:15:44.440874+0000","last_undegraded":"2024-04-15T15:15:44.440874+0000","last_fullsized":"2024-04-15T15:15:44.440874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:07.747265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587697+0000","last_change":"2024-04-15T15:15:37.557027+0000","last_active":"2024-04-15T15:15:43.587697+0000","last_peered":"2024-04-15T15:15:43.587697+0000","last_clean":"2024-04-15T15:15:43.587697+0000","last_became_active":"2024-04-15T15:15:37.556725+0000","last_became_peered":"2024-04-15T15:15:37.556725+0000","last_unstale":"2024-04-15T15:15:43.587697+0000","last_undegraded":"2024-04-15T15:15:43.587697+0000","last_fullsized":"2024-04-15T15:15:43.587697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:31.128525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439555+0000","last_change":"2024-04-15T15:15:42.593049+0000","last_active":"2024-04-15T15:15:44.439555+0000","last_peered":"2024-04-15T15:15:44.439555+0000","last_clean":"2024-04-15T15:15:44.439555+0000","last_became_active":"2024-04-15T15:15:42.592494+0000","last_became_peered":"2024-04-15T15:15:42.592494+0000","last_unstale":"2024-04-15T15:15:44.439555+0000","last_undegraded":"2024-04-15T15:15:44.439555+0000","last_fullsized":"2024-04-15T15:15:44.439555+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440121+0000","last_change":"2024-04-15T15:15:38.574063+0000","last_active":"2024-04-15T15:15:44.440121+0000","last_peered":"2024-04-15T15:15:44.440121+0000","last_clean":"2024-04-15T15:15:44.440121+0000","last_became_active":"2024-04-15T15:15:38.572802+0000","last_became_peered":"2024-04-15T15:15:38.572802+0000","last_unstale":"2024-04-15T15:15:44.440121+0000","last_undegraded":"2024-04-15T15:15:44.440121+0000","last_fullsized":"2024-04-15T15:15:44.440121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:42.311668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769497+0000","last_change":"2024-04-15T15:15:37.563114+0000","last_active":"2024-04-15T15:15:43.769497+0000","last_peered":"2024-04-15T15:15:43.769497+0000","last_clean":"2024-04-15T15:15:43.769497+0000","last_became_active":"2024-04-15T15:15:37.562746+0000","last_became_peered":"2024-04-15T15:15:37.562746+0000","last_unstale":"2024-04-15T15:15:43.769497+0000","last_undegraded":"2024-04-15T15:15:43.769497+0000","last_fullsized":"2024-04-15T15:15:43.769497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439239+0000","last_change":"2024-04-15T15:15:42.658786+0000","last_active":"2024-04-15T15:15:44.439239+0000","last_peered":"2024-04-15T15:15:44.439239+0000","last_clean":"2024-04-15T15:15:44.439239+0000","last_became_active":"2024-04-15T15:15:42.658329+0000","last_became_peered":"2024-04-15T15:15:42.658329+0000","last_unstale":"2024-04-15T15:15:44.439239+0000","last_undegraded":"2024-04-15T15:15:44.439239+0000","last_fullsized":"2024-04-15T15:15:44.439239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439930+0000","last_change":"2024-04-15T15:15:38.573032+0000","last_active":"2024-04-15T15:15:44.439930+0000","last_peered":"2024-04-15T15:15:44.439930+0000","last_clean":"2024-04-15T15:15:44.439930+0000","last_became_active":"2024-04-15T15:15:38.572689+0000","last_became_peered":"2024-04-15T15:15:38.572689+0000","last_unstale":"2024-04-15T15:15:44.439930+0000","last_undegraded":"2024-04-15T15:15:44.439930+0000","last_fullsized":"2024-04-15T15:15:44.439930+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:29.443728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588533+0000","last_change":"2024-04-15T15:15:37.565326+0000","last_active":"2024-04-15T15:15:43.588533+0000","last_peered":"2024-04-15T15:15:43.588533+0000","last_clean":"2024-04-15T15:15:43.588533+0000","last_became_active":"2024-04-15T15:15:37.564993+0000","last_became_peered":"2024-04-15T15:15:37.564993+0000","last_unstale":"2024-04-15T15:15:43.588533+0000","last_undegraded":"2024-04-15T15:15:43.588533+0000","last_fullsized":"2024-04-15T15:15:43.588533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:03:33.287332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588553+0000","last_change":"2024-04-15T15:15:42.590642+0000","last_active":"2024-04-15T15:15:43.588553+0000","last_peered":"2024-04-15T15:15:43.588553+0000","last_clean":"2024-04-15T15:15:43.588553+0000","last_became_active":"2024-04-15T15:15:42.590216+0000","last_became_peered":"2024-04-15T15:15:42.590216+0000","last_unstale":"2024-04-15T15:15:43.588553+0000","last_undegraded":"2024-04-15T15:15:43.588553+0000","last_fullsized":"2024-04-15T15:15:43.588553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:11.772474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439674+0000","last_change":"2024-04-15T15:15:38.571884+0000","last_active":"2024-04-15T15:15:44.439674+0000","last_peered":"2024-04-15T15:15:44.439674+0000","last_clean":"2024-04-15T15:15:44.439674+0000","last_became_active":"2024-04-15T15:15:38.571574+0000","last_became_peered":"2024-04-15T15:15:38.571574+0000","last_unstale":"2024-04-15T15:15:44.439674+0000","last_undegraded":"2024-04-15T15:15:44.439674+0000","last_fullsized":"2024-04-15T15:15:44.439674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769091+0000","last_change":"2024-04-15T15:15:37.558157+0000","last_active":"2024-04-15T15:15:43.769091+0000","last_peered":"2024-04-15T15:15:43.769091+0000","last_clean":"2024-04-15T15:15:43.769091+0000","last_became_active":"2024-04-15T15:15:37.557622+0000","last_became_peered":"2024-04-15T15:15:37.557622+0000","last_unstale":"2024-04-15T15:15:43.769091+0000","last_undegraded":"2024-04-15T15:15:43.769091+0000","last_fullsized":"2024-04-15T15:15:43.769091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439409+0000","last_change":"2024-04-15T15:15:42.589002+0000","last_active":"2024-04-15T15:15:44.439409+0000","last_peered":"2024-04-15T15:15:44.439409+0000","last_clean":"2024-04-15T15:15:44.439409+0000","last_became_active":"2024-04-15T15:15:42.588573+0000","last_became_peered":"2024-04-15T15:15:42.588573+0000","last_unstale":"2024-04-15T15:15:44.439409+0000","last_undegraded":"2024-04-15T15:15:44.439409+0000","last_fullsized":"2024-04-15T15:15:44.439409+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:08.525286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588824+0000","last_change":"2024-04-15T15:15:38.565226+0000","last_active":"2024-04-15T15:15:43.588824+0000","last_peered":"2024-04-15T15:15:43.588824+0000","last_clean":"2024-04-15T15:15:43.588824+0000","last_became_active":"2024-04-15T15:15:38.564938+0000","last_became_peered":"2024-04-15T15:15:38.564938+0000","last_unstale":"2024-04-15T15:15:43.588824+0000","last_undegraded":"2024-04-15T15:15:43.588824+0000","last_fullsized":"2024-04-15T15:15:43.588824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:54.407905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587872+0000","last_change":"2024-04-15T15:15:37.560743+0000","last_active":"2024-04-15T15:15:43.587872+0000","last_peered":"2024-04-15T15:15:43.587872+0000","last_clean":"2024-04-15T15:15:43.587872+0000","last_became_active":"2024-04-15T15:15:37.560439+0000","last_became_peered":"2024-04-15T15:15:37.560439+0000","last_unstale":"2024-04-15T15:15:43.587872+0000","last_undegraded":"2024-04-15T15:15:43.587872+0000","last_fullsized":"2024-04-15T15:15:43.587872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:33:51.276441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440270+0000","last_change":"2024-04-15T15:15:42.660245+0000","last_active":"2024-04-15T15:15:44.440270+0000","last_peered":"2024-04-15T15:15:44.440270+0000","last_clean":"2024-04-15T15:15:44.440270+0000","last_became_active":"2024-04-15T15:15:42.659882+0000","last_became_peered":"2024-04-15T15:15:42.659882+0000","last_unstale":"2024-04-15T15:15:44.440270+0000","last_undegraded":"2024-04-15T15:15:44.440270+0000","last_fullsized":"2024-04-15T15:15:44.440270+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:16:55.693482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440094+0000","last_change":"2024-04-15T15:15:38.572105+0000","last_active":"2024-04-15T15:15:44.440094+0000","last_peered":"2024-04-15T15:15:44.440094+0000","last_clean":"2024-04-15T15:15:44.440094+0000","last_became_active":"2024-04-15T15:15:38.571821+0000","last_became_peered":"2024-04-15T15:15:38.571821+0000","last_unstale":"2024-04-15T15:15:44.440094+0000","last_undegraded":"2024-04-15T15:15:44.440094+0000","last_fullsized":"2024-04-15T15:15:44.440094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440332+0000","last_change":"2024-04-15T15:15:37.562506+0000","last_active":"2024-04-15T15:15:44.440332+0000","last_peered":"2024-04-15T15:15:44.440332+0000","last_clean":"2024-04-15T15:15:44.440332+0000","last_became_active":"2024-04-15T15:15:37.562212+0000","last_became_peered":"2024-04-15T15:15:37.562212+0000","last_unstale":"2024-04-15T15:15:44.440332+0000","last_undegraded":"2024-04-15T15:15:44.440332+0000","last_fullsized":"2024-04-15T15:15:44.440332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440199+0000","last_change":"2024-04-15T15:15:42.656037+0000","last_active":"2024-04-15T15:15:44.440199+0000","last_peered":"2024-04-15T15:15:44.440199+0000","last_clean":"2024-04-15T15:15:44.440199+0000","last_became_active":"2024-04-15T15:15:42.655647+0000","last_became_peered":"2024-04-15T15:15:42.655647+0000","last_unstale":"2024-04-15T15:15:44.440199+0000","last_undegraded":"2024-04-15T15:15:44.440199+0000","last_fullsized":"2024-04-15T15:15:44.440199+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:53:58.204060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439284+0000","last_change":"2024-04-15T15:15:38.573064+0000","last_active":"2024-04-15T15:15:44.439284+0000","last_peered":"2024-04-15T15:15:44.439284+0000","last_clean":"2024-04-15T15:15:44.439284+0000","last_became_active":"2024-04-15T15:15:38.572766+0000","last_became_peered":"2024-04-15T15:15:38.572766+0000","last_unstale":"2024-04-15T15:15:44.439284+0000","last_undegraded":"2024-04-15T15:15:44.439284+0000","last_fullsized":"2024-04-15T15:15:44.439284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:53:30.455027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439611+0000","last_change":"2024-04-15T15:15:37.565383+0000","last_active":"2024-04-15T15:15:44.439611+0000","last_peered":"2024-04-15T15:15:44.439611+0000","last_clean":"2024-04-15T15:15:44.439611+0000","last_became_active":"2024-04-15T15:15:37.561007+0000","last_became_peered":"2024-04-15T15:15:37.561007+0000","last_unstale":"2024-04-15T15:15:44.439611+0000","last_undegraded":"2024-04-15T15:15:44.439611+0000","last_fullsized":"2024-04-15T15:15:44.439611+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440266+0000","last_change":"2024-04-15T15:15:42.657177+0000","last_active":"2024-04-15T15:15:44.440266+0000","last_peered":"2024-04-15T15:15:44.440266+0000","last_clean":"2024-04-15T15:15:44.440266+0000","last_became_active":"2024-04-15T15:15:42.656776+0000","last_became_peered":"2024-04-15T15:15:42.656776+0000","last_unstale":"2024-04-15T15:15:44.440266+0000","last_undegraded":"2024-04-15T15:15:44.440266+0000","last_fullsized":"2024-04-15T15:15:44.440266+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:04.807386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439546+0000","last_change":"2024-04-15T15:15:38.569532+0000","last_active":"2024-04-15T15:15:44.439546+0000","last_peered":"2024-04-15T15:15:44.439546+0000","last_clean":"2024-04-15T15:15:44.439546+0000","last_became_active":"2024-04-15T15:15:38.569097+0000","last_became_peered":"2024-04-15T15:15:38.569097+0000","last_unstale":"2024-04-15T15:15:44.439546+0000","last_undegraded":"2024-04-15T15:15:44.439546+0000","last_fullsized":"2024-04-15T15:15:44.439546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:48.573127+0000","last_change":"2024-04-15T15:15:37.559420+0000","last_active":"2024-04-15T15:15:48.573127+0000","last_peered":"2024-04-15T15:15:48.573127+0000","last_clean":"2024-04-15T15:15:48.573127+0000","last_became_active":"2024-04-15T15:15:37.559152+0000","last_became_peered":"2024-04-15T15:15:37.559152+0000","last_unstale":"2024-04-15T15:15:48.573127+0000","last_undegraded":"2024-04-15T15:15:48.573127+0000","last_fullsized":"2024-04-15T15:15:48.573127+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:56:49.423178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587812+0000","last_change":"2024-04-15T15:15:42.591766+0000","last_active":"2024-04-15T15:15:43.587812+0000","last_peered":"2024-04-15T15:15:43.587812+0000","last_clean":"2024-04-15T15:15:43.587812+0000","last_became_active":"2024-04-15T15:15:42.591354+0000","last_became_peered":"2024-04-15T15:15:42.591354+0000","last_unstale":"2024-04-15T15:15:43.587812+0000","last_undegraded":"2024-04-15T15:15:43.587812+0000","last_fullsized":"2024-04-15T15:15:43.587812+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:45.129902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588450+0000","last_change":"2024-04-15T15:15:38.562820+0000","last_active":"2024-04-15T15:15:43.588450+0000","last_peered":"2024-04-15T15:15:43.588450+0000","last_clean":"2024-04-15T15:15:43.588450+0000","last_became_active":"2024-04-15T15:15:38.562536+0000","last_became_peered":"2024-04-15T15:15:38.562536+0000","last_unstale":"2024-04-15T15:15:43.588450+0000","last_undegraded":"2024-04-15T15:15:43.588450+0000","last_fullsized":"2024-04-15T15:15:43.588450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587577+0000","last_change":"2024-04-15T15:15:37.558919+0000","last_active":"2024-04-15T15:15:43.587577+0000","last_peered":"2024-04-15T15:15:43.587577+0000","last_clean":"2024-04-15T15:15:43.587577+0000","last_became_active":"2024-04-15T15:15:37.558592+0000","last_became_peered":"2024-04-15T15:15:37.558592+0000","last_unstale":"2024-04-15T15:15:43.587577+0000","last_undegraded":"2024-04-15T15:15:43.587577+0000","last_fullsized":"2024-04-15T15:15:43.587577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:29.931213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587908+0000","last_change":"2024-04-15T15:15:42.591079+0000","last_active":"2024-04-15T15:15:43.587908+0000","last_peered":"2024-04-15T15:15:43.587908+0000","last_clean":"2024-04-15T15:15:43.587908+0000","last_became_active":"2024-04-15T15:15:42.590633+0000","last_became_peered":"2024-04-15T15:15:42.590633+0000","last_unstale":"2024-04-15T15:15:43.587908+0000","last_undegraded":"2024-04-15T15:15:43.587908+0000","last_fullsized":"2024-04-15T15:15:43.587908+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:33.636220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439372+0000","last_change":"2024-04-15T15:15:38.574069+0000","last_active":"2024-04-15T15:15:44.439372+0000","last_peered":"2024-04-15T15:15:44.439372+0000","last_clean":"2024-04-15T15:15:44.439372+0000","last_became_active":"2024-04-15T15:15:38.573139+0000","last_became_peered":"2024-04-15T15:15:38.573139+0000","last_unstale":"2024-04-15T15:15:44.439372+0000","last_undegraded":"2024-04-15T15:15:44.439372+0000","last_fullsized":"2024-04-15T15:15:44.439372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440496+0000","last_change":"2024-04-15T15:15:37.563424+0000","last_active":"2024-04-15T15:15:44.440496+0000","last_peered":"2024-04-15T15:15:44.440496+0000","last_clean":"2024-04-15T15:15:44.440496+0000","last_became_active":"2024-04-15T15:15:37.562924+0000","last_became_peered":"2024-04-15T15:15:37.562924+0000","last_unstale":"2024-04-15T15:15:44.440496+0000","last_undegraded":"2024-04-15T15:15:44.440496+0000","last_fullsized":"2024-04-15T15:15:44.440496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439129+0000","last_change":"2024-04-15T15:15:42.587632+0000","last_active":"2024-04-15T15:15:44.439129+0000","last_peered":"2024-04-15T15:15:44.439129+0000","last_clean":"2024-04-15T15:15:44.439129+0000","last_became_active":"2024-04-15T15:15:42.587139+0000","last_became_peered":"2024-04-15T15:15:42.587139+0000","last_unstale":"2024-04-15T15:15:44.439129+0000","last_undegraded":"2024-04-15T15:15:44.439129+0000","last_fullsized":"2024-04-15T15:15:44.439129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:17.244528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440397+0000","last_change":"2024-04-15T15:15:38.567493+0000","last_active":"2024-04-15T15:15:44.440397+0000","last_peered":"2024-04-15T15:15:44.440397+0000","last_clean":"2024-04-15T15:15:44.440397+0000","last_became_active":"2024-04-15T15:15:38.567188+0000","last_became_peered":"2024-04-15T15:15:38.567188+0000","last_unstale":"2024-04-15T15:15:44.440397+0000","last_undegraded":"2024-04-15T15:15:44.440397+0000","last_fullsized":"2024-04-15T15:15:44.440397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588898+0000","last_change":"2024-04-15T15:15:37.563986+0000","last_active":"2024-04-15T15:15:43.588898+0000","last_peered":"2024-04-15T15:15:43.588898+0000","last_clean":"2024-04-15T15:15:43.588898+0000","last_became_active":"2024-04-15T15:15:37.563666+0000","last_became_peered":"2024-04-15T15:15:37.563666+0000","last_unstale":"2024-04-15T15:15:43.588898+0000","last_undegraded":"2024-04-15T15:15:43.588898+0000","last_fullsized":"2024-04-15T15:15:43.588898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:33:07.645968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440053+0000","last_change":"2024-04-15T15:15:42.661474+0000","last_active":"2024-04-15T15:15:44.440053+0000","last_peered":"2024-04-15T15:15:44.440053+0000","last_clean":"2024-04-15T15:15:44.440053+0000","last_became_active":"2024-04-15T15:15:42.660740+0000","last_became_peered":"2024-04-15T15:15:42.660740+0000","last_unstale":"2024-04-15T15:15:44.440053+0000","last_undegraded":"2024-04-15T15:15:44.440053+0000","last_fullsized":"2024-04-15T15:15:44.440053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:56:48.355956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439422+0000","last_change":"2024-04-15T15:15:42.658445+0000","last_active":"2024-04-15T15:15:44.439422+0000","last_peered":"2024-04-15T15:15:44.439422+0000","last_clean":"2024-04-15T15:15:44.439422+0000","last_became_active":"2024-04-15T15:15:42.658104+0000","last_became_peered":"2024-04-15T15:15:42.658104+0000","last_unstale":"2024-04-15T15:15:44.439422+0000","last_undegraded":"2024-04-15T15:15:44.439422+0000","last_fullsized":"2024-04-15T15:15:44.439422+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:55:33.468937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586589+0000","last_change":"2024-04-15T15:15:38.562893+0000","last_active":"2024-04-15T15:15:43.586589+0000","last_peered":"2024-04-15T15:15:43.586589+0000","last_clean":"2024-04-15T15:15:43.586589+0000","last_became_active":"2024-04-15T15:15:38.562604+0000","last_became_peered":"2024-04-15T15:15:38.562604+0000","last_unstale":"2024-04-15T15:15:43.586589+0000","last_undegraded":"2024-04-15T15:15:43.586589+0000","last_fullsized":"2024-04-15T15:15:43.586589+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439406+0000","last_change":"2024-04-15T15:15:37.556524+0000","last_active":"2024-04-15T15:15:44.439406+0000","last_peered":"2024-04-15T15:15:44.439406+0000","last_clean":"2024-04-15T15:15:44.439406+0000","last_became_active":"2024-04-15T15:15:37.556249+0000","last_became_peered":"2024-04-15T15:15:37.556249+0000","last_unstale":"2024-04-15T15:15:44.439406+0000","last_undegraded":"2024-04-15T15:15:44.439406+0000","last_fullsized":"2024-04-15T15:15:44.439406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:08.649186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588777+0000","last_change":"2024-04-15T15:15:42.590110+0000","last_active":"2024-04-15T15:15:43.588777+0000","last_peered":"2024-04-15T15:15:43.588777+0000","last_clean":"2024-04-15T15:15:43.588777+0000","last_became_active":"2024-04-15T15:15:42.589355+0000","last_became_peered":"2024-04-15T15:15:42.589355+0000","last_unstale":"2024-04-15T15:15:43.588777+0000","last_undegraded":"2024-04-15T15:15:43.588777+0000","last_fullsized":"2024-04-15T15:15:43.588777+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:14.096399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439180+0000","last_change":"2024-04-15T15:15:38.567772+0000","last_active":"2024-04-15T15:15:44.439180+0000","last_peered":"2024-04-15T15:15:44.439180+0000","last_clean":"2024-04-15T15:15:44.439180+0000","last_became_active":"2024-04-15T15:15:38.566987+0000","last_became_peered":"2024-04-15T15:15:38.566987+0000","last_unstale":"2024-04-15T15:15:44.439180+0000","last_undegraded":"2024-04-15T15:15:44.439180+0000","last_fullsized":"2024-04-15T15:15:44.439180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440394+0000","last_change":"2024-04-15T15:15:37.561996+0000","last_active":"2024-04-15T15:15:44.440394+0000","last_peered":"2024-04-15T15:15:44.440394+0000","last_clean":"2024-04-15T15:15:44.440394+0000","last_became_active":"2024-04-15T15:15:37.561696+0000","last_became_peered":"2024-04-15T15:15:37.561696+0000","last_unstale":"2024-04-15T15:15:44.440394+0000","last_undegraded":"2024-04-15T15:15:44.440394+0000","last_fullsized":"2024-04-15T15:15:44.440394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:10:18.866169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588032+0000","last_change":"2024-04-15T15:15:37.561778+0000","last_active":"2024-04-15T15:15:43.588032+0000","last_peered":"2024-04-15T15:15:43.588032+0000","last_clean":"2024-04-15T15:15:43.588032+0000","last_became_active":"2024-04-15T15:15:37.561489+0000","last_became_peered":"2024-04-15T15:15:37.561489+0000","last_unstale":"2024-04-15T15:15:43.588032+0000","last_undegraded":"2024-04-15T15:15:43.588032+0000","last_fullsized":"2024-04-15T15:15:43.588032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:55.005069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439557+0000","last_change":"2024-04-15T15:15:38.570137+0000","last_active":"2024-04-15T15:15:44.439557+0000","last_peered":"2024-04-15T15:15:44.439557+0000","last_clean":"2024-04-15T15:15:44.439557+0000","last_became_active":"2024-04-15T15:15:38.569657+0000","last_became_peered":"2024-04-15T15:15:38.569657+0000","last_unstale":"2024-04-15T15:15:44.439557+0000","last_undegraded":"2024-04-15T15:15:44.439557+0000","last_fullsized":"2024-04-15T15:15:44.439557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769366+0000","last_change":"2024-04-15T15:15:42.584680+0000","last_active":"2024-04-15T15:15:43.769366+0000","last_peered":"2024-04-15T15:15:43.769366+0000","last_clean":"2024-04-15T15:15:43.769366+0000","last_became_active":"2024-04-15T15:15:42.584312+0000","last_became_peered":"2024-04-15T15:15:42.584312+0000","last_unstale":"2024-04-15T15:15:43.769366+0000","last_undegraded":"2024-04-15T15:15:43.769366+0000","last_fullsized":"2024-04-15T15:15:43.769366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:39.473475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440046+0000","last_change":"2024-04-15T15:15:37.565397+0000","last_active":"2024-04-15T15:15:44.440046+0000","last_peered":"2024-04-15T15:15:44.440046+0000","last_clean":"2024-04-15T15:15:44.440046+0000","last_became_active":"2024-04-15T15:15:37.565018+0000","last_became_peered":"2024-04-15T15:15:37.565018+0000","last_unstale":"2024-04-15T15:15:44.440046+0000","last_undegraded":"2024-04-15T15:15:44.440046+0000","last_fullsized":"2024-04-15T15:15:44.440046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:36.534540+0000","last_clean_scrub_stamp":"2024-04-15T15:15:36.534540+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:02:14.346151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587469+0000","last_change":"2024-04-15T15:15:38.558591+0000","last_active":"2024-04-15T15:15:43.587469+0000","last_peered":"2024-04-15T15:15:43.587469+0000","last_clean":"2024-04-15T15:15:43.587469+0000","last_became_active":"2024-04-15T15:15:38.558044+0000","last_became_peered":"2024-04-15T15:15:38.558044+0000","last_unstale":"2024-04-15T15:15:43.587469+0000","last_undegraded":"2024-04-15T15:15:43.587469+0000","last_fullsized":"2024-04-15T15:15:43.587469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:35:56.009007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439916+0000","last_change":"2024-04-15T15:15:42.587509+0000","last_active":"2024-04-15T15:15:44.439916+0000","last_peered":"2024-04-15T15:15:44.439916+0000","last_clean":"2024-04-15T15:15:44.439916+0000","last_became_active":"2024-04-15T15:15:42.587097+0000","last_became_peered":"2024-04-15T15:15:42.587097+0000","last_unstale":"2024-04-15T15:15:44.439916+0000","last_undegraded":"2024-04-15T15:15:44.439916+0000","last_fullsized":"2024-04-15T15:15:44.439916+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:41.563688+0000","last_clean_scrub_stamp":"2024-04-15T15:15:41.563688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:55:01.099477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588741+0000","last_change":"2024-04-15T15:15:38.566026+0000","last_active":"2024-04-15T15:15:43.588741+0000","last_peered":"2024-04-15T15:15:43.588741+0000","last_clean":"2024-04-15T15:15:43.588741+0000","last_became_active":"2024-04-15T15:15:38.565726+0000","last_became_peered":"2024-04-15T15:15:38.565726+0000","last_unstale":"2024-04-15T15:15:43.588741+0000","last_undegraded":"2024-04-15T15:15:43.588741+0000","last_fullsized":"2024-04-15T15:15:43.588741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769196+0000","last_change":"2024-04-15T15:15:38.567487+0000","last_active":"2024-04-15T15:15:43.769196+0000","last_peered":"2024-04-15T15:15:43.769196+0000","last_clean":"2024-04-15T15:15:43.769196+0000","last_became_active":"2024-04-15T15:15:38.567201+0000","last_became_peered":"2024-04-15T15:15:38.567201+0000","last_unstale":"2024-04-15T15:15:43.769196+0000","last_undegraded":"2024-04-15T15:15:43.769196+0000","last_fullsized":"2024-04-15T15:15:43.769196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:37:32.132463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440239+0000","last_change":"2024-04-15T15:15:38.569794+0000","last_active":"2024-04-15T15:15:44.440239+0000","last_peered":"2024-04-15T15:15:44.440239+0000","last_clean":"2024-04-15T15:15:44.440239+0000","last_became_active":"2024-04-15T15:15:38.569510+0000","last_became_peered":"2024-04-15T15:15:38.569510+0000","last_unstale":"2024-04-15T15:15:44.440239+0000","last_undegraded":"2024-04-15T15:15:44.440239+0000","last_fullsized":"2024-04-15T15:15:44.440239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:48:29.611010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439457+0000","last_change":"2024-04-15T15:15:38.563215+0000","last_active":"2024-04-15T15:15:44.439457+0000","last_peered":"2024-04-15T15:15:44.439457+0000","last_clean":"2024-04-15T15:15:44.439457+0000","last_became_active":"2024-04-15T15:15:38.562942+0000","last_became_peered":"2024-04-15T15:15:38.562942+0000","last_unstale":"2024-04-15T15:15:44.439457+0000","last_undegraded":"2024-04-15T15:15:44.439457+0000","last_fullsized":"2024-04-15T15:15:44.439457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440355+0000","last_change":"2024-04-15T15:15:38.567774+0000","last_active":"2024-04-15T15:15:44.440355+0000","last_peered":"2024-04-15T15:15:44.440355+0000","last_clean":"2024-04-15T15:15:44.440355+0000","last_became_active":"2024-04-15T15:15:38.567471+0000","last_became_peered":"2024-04-15T15:15:38.567471+0000","last_unstale":"2024-04-15T15:15:44.440355+0000","last_undegraded":"2024-04-15T15:15:44.440355+0000","last_fullsized":"2024-04-15T15:15:44.440355+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587284+0000","last_change":"2024-04-15T15:15:38.562963+0000","last_active":"2024-04-15T15:15:43.587284+0000","last_peered":"2024-04-15T15:15:43.587284+0000","last_clean":"2024-04-15T15:15:43.587284+0000","last_became_active":"2024-04-15T15:15:38.562611+0000","last_became_peered":"2024-04-15T15:15:38.562611+0000","last_unstale":"2024-04-15T15:15:43.587284+0000","last_undegraded":"2024-04-15T15:15:43.587284+0000","last_fullsized":"2024-04-15T15:15:43.587284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588859+0000","last_change":"2024-04-15T15:15:38.566158+0000","last_active":"2024-04-15T15:15:43.588859+0000","last_peered":"2024-04-15T15:15:43.588859+0000","last_clean":"2024-04-15T15:15:43.588859+0000","last_became_active":"2024-04-15T15:15:38.565859+0000","last_became_peered":"2024-04-15T15:15:38.565859+0000","last_unstale":"2024-04-15T15:15:43.588859+0000","last_undegraded":"2024-04-15T15:15:43.588859+0000","last_fullsized":"2024-04-15T15:15:43.588859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440318+0000","last_change":"2024-04-15T15:15:38.566715+0000","last_active":"2024-04-15T15:15:44.440318+0000","last_peered":"2024-04-15T15:15:44.440318+0000","last_clean":"2024-04-15T15:15:44.440318+0000","last_became_active":"2024-04-15T15:15:38.565677+0000","last_became_peered":"2024-04-15T15:15:38.565677+0000","last_unstale":"2024-04-15T15:15:44.440318+0000","last_undegraded":"2024-04-15T15:15:44.440318+0000","last_fullsized":"2024-04-15T15:15:44.440318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439302+0000","last_change":"2024-04-15T15:15:38.565562+0000","last_active":"2024-04-15T15:15:44.439302+0000","last_peered":"2024-04-15T15:15:44.439302+0000","last_clean":"2024-04-15T15:15:44.439302+0000","last_became_active":"2024-04-15T15:15:38.564779+0000","last_became_peered":"2024-04-15T15:15:38.564779+0000","last_unstale":"2024-04-15T15:15:44.439302+0000","last_undegraded":"2024-04-15T15:15:44.439302+0000","last_fullsized":"2024-04-15T15:15:44.439302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:30:10.071190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440302+0000","last_change":"2024-04-15T15:15:38.557876+0000","last_active":"2024-04-15T15:15:44.440302+0000","last_peered":"2024-04-15T15:15:44.440302+0000","last_clean":"2024-04-15T15:15:44.440302+0000","last_became_active":"2024-04-15T15:15:38.557568+0000","last_became_peered":"2024-04-15T15:15:38.557568+0000","last_unstale":"2024-04-15T15:15:44.440302+0000","last_undegraded":"2024-04-15T15:15:44.440302+0000","last_fullsized":"2024-04-15T15:15:44.440302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440139+0000","last_change":"2024-04-15T15:15:38.568437+0000","last_active":"2024-04-15T15:15:44.440139+0000","last_peered":"2024-04-15T15:15:44.440139+0000","last_clean":"2024-04-15T15:15:44.440139+0000","last_became_active":"2024-04-15T15:15:38.567071+0000","last_became_peered":"2024-04-15T15:15:38.567071+0000","last_unstale":"2024-04-15T15:15:44.440139+0000","last_undegraded":"2024-04-15T15:15:44.440139+0000","last_fullsized":"2024-04-15T15:15:44.440139+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:41:37.869359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439337+0000","last_change":"2024-04-15T15:15:38.572158+0000","last_active":"2024-04-15T15:15:44.439337+0000","last_peered":"2024-04-15T15:15:44.439337+0000","last_clean":"2024-04-15T15:15:44.439337+0000","last_became_active":"2024-04-15T15:15:38.571868+0000","last_became_peered":"2024-04-15T15:15:38.571868+0000","last_unstale":"2024-04-15T15:15:44.439337+0000","last_undegraded":"2024-04-15T15:15:44.439337+0000","last_fullsized":"2024-04-15T15:15:44.439337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:20:11.341387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588528+0000","last_change":"2024-04-15T15:15:38.556771+0000","last_active":"2024-04-15T15:15:43.588528+0000","last_peered":"2024-04-15T15:15:43.588528+0000","last_clean":"2024-04-15T15:15:43.588528+0000","last_became_active":"2024-04-15T15:15:38.556456+0000","last_became_peered":"2024-04-15T15:15:38.556456+0000","last_unstale":"2024-04-15T15:15:43.588528+0000","last_undegraded":"2024-04-15T15:15:43.588528+0000","last_fullsized":"2024-04-15T15:15:43.588528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439982+0000","last_change":"2024-04-15T15:15:38.565285+0000","last_active":"2024-04-15T15:15:44.439982+0000","last_peered":"2024-04-15T15:15:44.439982+0000","last_clean":"2024-04-15T15:15:44.439982+0000","last_became_active":"2024-04-15T15:15:38.564505+0000","last_became_peered":"2024-04-15T15:15:38.564505+0000","last_unstale":"2024-04-15T15:15:44.439982+0000","last_undegraded":"2024-04-15T15:15:44.439982+0000","last_fullsized":"2024-04-15T15:15:44.439982+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:13:44.185650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.769488+0000","last_change":"2024-04-15T15:15:38.562279+0000","last_active":"2024-04-15T15:15:43.769488+0000","last_peered":"2024-04-15T15:15:43.769488+0000","last_clean":"2024-04-15T15:15:43.769488+0000","last_became_active":"2024-04-15T15:15:38.561971+0000","last_became_peered":"2024-04-15T15:15:38.561971+0000","last_unstale":"2024-04-15T15:15:43.769488+0000","last_undegraded":"2024-04-15T15:15:43.769488+0000","last_fullsized":"2024-04-15T15:15:43.769488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440908+0000","last_change":"2024-04-15T15:15:38.569181+0000","last_active":"2024-04-15T15:15:44.440908+0000","last_peered":"2024-04-15T15:15:44.440908+0000","last_clean":"2024-04-15T15:15:44.440908+0000","last_became_active":"2024-04-15T15:15:38.566602+0000","last_became_peered":"2024-04-15T15:15:38.566602+0000","last_unstale":"2024-04-15T15:15:44.440908+0000","last_undegraded":"2024-04-15T15:15:44.440908+0000","last_fullsized":"2024-04-15T15:15:44.440908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:49:58.773063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439191+0000","last_change":"2024-04-15T15:15:38.568218+0000","last_active":"2024-04-15T15:15:44.439191+0000","last_peered":"2024-04-15T15:15:44.439191+0000","last_clean":"2024-04-15T15:15:44.439191+0000","last_became_active":"2024-04-15T15:15:38.566966+0000","last_became_peered":"2024-04-15T15:15:38.566966+0000","last_unstale":"2024-04-15T15:15:44.439191+0000","last_undegraded":"2024-04-15T15:15:44.439191+0000","last_fullsized":"2024-04-15T15:15:44.439191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.586673+0000","last_change":"2024-04-15T15:15:38.563656+0000","last_active":"2024-04-15T15:15:43.586673+0000","last_peered":"2024-04-15T15:15:43.586673+0000","last_clean":"2024-04-15T15:15:43.586673+0000","last_became_active":"2024-04-15T15:15:38.563380+0000","last_became_peered":"2024-04-15T15:15:38.563380+0000","last_unstale":"2024-04-15T15:15:43.586673+0000","last_undegraded":"2024-04-15T15:15:43.586673+0000","last_fullsized":"2024-04-15T15:15:43.586673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:23:38.169765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439190+0000","last_change":"2024-04-15T15:15:38.558804+0000","last_active":"2024-04-15T15:15:44.439190+0000","last_peered":"2024-04-15T15:15:44.439190+0000","last_clean":"2024-04-15T15:15:44.439190+0000","last_became_active":"2024-04-15T15:15:38.558374+0000","last_became_peered":"2024-04-15T15:15:38.558374+0000","last_unstale":"2024-04-15T15:15:44.439190+0000","last_undegraded":"2024-04-15T15:15:44.439190+0000","last_fullsized":"2024-04-15T15:15:44.439190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:20:35.448266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439861+0000","last_change":"2024-04-15T15:15:38.564455+0000","last_active":"2024-04-15T15:15:44.439861+0000","last_peered":"2024-04-15T15:15:44.439861+0000","last_clean":"2024-04-15T15:15:44.439861+0000","last_became_active":"2024-04-15T15:15:38.563328+0000","last_became_peered":"2024-04-15T15:15:38.563328+0000","last_unstale":"2024-04-15T15:15:44.439861+0000","last_undegraded":"2024-04-15T15:15:44.439861+0000","last_fullsized":"2024-04-15T15:15:44.439861+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588424+0000","last_change":"2024-04-15T15:15:38.563202+0000","last_active":"2024-04-15T15:15:43.588424+0000","last_peered":"2024-04-15T15:15:43.588424+0000","last_clean":"2024-04-15T15:15:43.588424+0000","last_became_active":"2024-04-15T15:15:38.562862+0000","last_became_peered":"2024-04-15T15:15:38.562862+0000","last_unstale":"2024-04-15T15:15:43.588424+0000","last_undegraded":"2024-04-15T15:15:43.588424+0000","last_fullsized":"2024-04-15T15:15:43.588424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:39:30.935152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588478+0000","last_change":"2024-04-15T15:15:38.570615+0000","last_active":"2024-04-15T15:15:43.588478+0000","last_peered":"2024-04-15T15:15:43.588478+0000","last_clean":"2024-04-15T15:15:43.588478+0000","last_became_active":"2024-04-15T15:15:38.570340+0000","last_became_peered":"2024-04-15T15:15:38.570340+0000","last_unstale":"2024-04-15T15:15:43.588478+0000","last_undegraded":"2024-04-15T15:15:43.588478+0000","last_fullsized":"2024-04-15T15:15:43.588478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:12.585986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.588803+0000","last_change":"2024-04-15T15:15:38.567756+0000","last_active":"2024-04-15T15:15:43.588803+0000","last_peered":"2024-04-15T15:15:43.588803+0000","last_clean":"2024-04-15T15:15:43.588803+0000","last_became_active":"2024-04-15T15:15:38.567455+0000","last_became_peered":"2024-04-15T15:15:38.567455+0000","last_unstale":"2024-04-15T15:15:43.588803+0000","last_undegraded":"2024-04-15T15:15:43.588803+0000","last_fullsized":"2024-04-15T15:15:43.588803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:17:06.893603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440004+0000","last_change":"2024-04-15T15:15:38.574989+0000","last_active":"2024-04-15T15:15:44.440004+0000","last_peered":"2024-04-15T15:15:44.440004+0000","last_clean":"2024-04-15T15:15:44.440004+0000","last_became_active":"2024-04-15T15:15:38.573776+0000","last_became_peered":"2024-04-15T15:15:38.573776+0000","last_unstale":"2024-04-15T15:15:44.440004+0000","last_undegraded":"2024-04-15T15:15:44.440004+0000","last_fullsized":"2024-04-15T15:15:44.440004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:44:41.104693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.438915+0000","last_change":"2024-04-15T15:15:38.573020+0000","last_active":"2024-04-15T15:15:44.438915+0000","last_peered":"2024-04-15T15:15:44.438915+0000","last_clean":"2024-04-15T15:15:44.438915+0000","last_became_active":"2024-04-15T15:15:38.572715+0000","last_became_peered":"2024-04-15T15:15:38.572715+0000","last_unstale":"2024-04-15T15:15:44.438915+0000","last_undegraded":"2024-04-15T15:15:44.438915+0000","last_fullsized":"2024-04-15T15:15:44.438915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439587+0000","last_change":"2024-04-15T15:15:38.568749+0000","last_active":"2024-04-15T15:15:44.439587+0000","last_peered":"2024-04-15T15:15:44.439587+0000","last_clean":"2024-04-15T15:15:44.439587+0000","last_became_active":"2024-04-15T15:15:38.568488+0000","last_became_peered":"2024-04-15T15:15:38.568488+0000","last_unstale":"2024-04-15T15:15:44.439587+0000","last_undegraded":"2024-04-15T15:15:44.439587+0000","last_fullsized":"2024-04-15T15:15:44.439587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:12:15.343632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440025+0000","last_change":"2024-04-15T15:15:38.572663+0000","last_active":"2024-04-15T15:15:44.440025+0000","last_peered":"2024-04-15T15:15:44.440025+0000","last_clean":"2024-04-15T15:15:44.440025+0000","last_became_active":"2024-04-15T15:15:38.572349+0000","last_became_peered":"2024-04-15T15:15:38.572349+0000","last_unstale":"2024-04-15T15:15:44.440025+0000","last_undegraded":"2024-04-15T15:15:44.440025+0000","last_fullsized":"2024-04-15T15:15:44.440025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:09:06.006871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439869+0000","last_change":"2024-04-15T15:15:38.567565+0000","last_active":"2024-04-15T15:15:44.439869+0000","last_peered":"2024-04-15T15:15:44.439869+0000","last_clean":"2024-04-15T15:15:44.439869+0000","last_became_active":"2024-04-15T15:15:38.567300+0000","last_became_peered":"2024-04-15T15:15:38.567300+0000","last_unstale":"2024-04-15T15:15:44.439869+0000","last_undegraded":"2024-04-15T15:15:44.439869+0000","last_fullsized":"2024-04-15T15:15:44.439869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:04:38.127457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":18,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.440752+0000","last_change":"2024-04-15T15:15:38.578049+0000","last_active":"2024-04-15T15:15:44.440752+0000","last_peered":"2024-04-15T15:15:44.440752+0000","last_clean":"2024-04-15T15:15:44.440752+0000","last_became_active":"2024-04-15T15:15:38.577801+0000","last_became_peered":"2024-04-15T15:15:38.577801+0000","last_unstale":"2024-04-15T15:15:44.440752+0000","last_undegraded":"2024-04-15T15:15:44.440752+0000","last_fullsized":"2024-04-15T15:15:44.440752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439206+0000","last_change":"2024-04-15T15:15:38.561841+0000","last_active":"2024-04-15T15:15:44.439206+0000","last_peered":"2024-04-15T15:15:44.439206+0000","last_clean":"2024-04-15T15:15:44.439206+0000","last_became_active":"2024-04-15T15:15:38.561554+0000","last_became_peered":"2024-04-15T15:15:38.561554+0000","last_unstale":"2024-04-15T15:15:44.439206+0000","last_undegraded":"2024-04-15T15:15:44.439206+0000","last_fullsized":"2024-04-15T15:15:44.439206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:03:49.101667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:43.587614+0000","last_change":"2024-04-15T15:15:38.560883+0000","last_active":"2024-04-15T15:15:43.587614+0000","last_peered":"2024-04-15T15:15:43.587614+0000","last_clean":"2024-04-15T15:15:43.587614+0000","last_became_active":"2024-04-15T15:15:38.560562+0000","last_became_peered":"2024-04-15T15:15:38.560562+0000","last_unstale":"2024-04-15T15:15:43.587614+0000","last_undegraded":"2024-04-15T15:15:43.587614+0000","last_fullsized":"2024-04-15T15:15:43.587614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:46.883381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:44.439744+0000","last_change":"2024-04-15T15:15:38.565038+0000","last_active":"2024-04-15T15:15:44.439744+0000","last_peered":"2024-04-15T15:15:44.439744+0000","last_clean":"2024-04-15T15:15:44.439744+0000","last_became_active":"2024-04-15T15:15:38.564551+0000","last_became_peered":"2024-04-15T15:15:38.564551+0000","last_unstale":"2024-04-15T15:15:44.439744+0000","last_undegraded":"2024-04-15T15:15:44.439744+0000","last_fullsized":"2024-04-15T15:15:44.439744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:37.538478+0000","last_clean_scrub_stamp":"2024-04-15T15:15:37.538478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:23:04.500077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542155,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":104177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542154,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":104213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":695037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":107929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":104087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":270336,"data_stored":104247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":868352,"data_stored":695093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:05.834 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:16:05.834 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T15:16:05.835 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-15T15:16:05.835 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-15T15:16:05.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.967+0000 7fd62c7a3640 1 Processor -- start 2024-04-15T15:16:05.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.967+0000 7fd62c7a3640 1 -- start start 2024-04-15T15:16:05.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.968+0000 7fd62c7a3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd62409b3e0 0x7fd6240b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.968+0000 7fd62c7a3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240d0490 con 0x7fd6240d0940 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.968+0000 7fd62c7a3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6240b1990 con 0x7fd62409ba10 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.968+0000 7fd62c7a3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd6240b1c70 con 0x7fd62409b3e0 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62ad40640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd62409b3e0 0x7fd6240b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62a53f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6240d0940 0x7fd6240d0d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55682/0 (socket says 172.21.15.46:55682) 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62a53f640 1 -- 172.21.15.46:0/2644866157 learned_addr learned my addr 172.21.15.46:0/2644866157 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62ad40640 1 -- 172.21.15.46:0/2644866157 >> v1:172.21.15.78:6789/0 conn(0x7fd62409ba10 legacy=0x7fd6240b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62ad40640 1 -- 172.21.15.46:0/2644866157 >> v1:172.21.15.46:6789/0 conn(0x7fd6240d0940 legacy=0x7fd6240d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62953d640 1 -- 172.21.15.46:0/2644866157 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660783293 0 0) 0x7fd614001490 con 0x7fd6240d0940 2024-04-15T15:16:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.969+0000 7fd62ad40640 1 -- 172.21.15.46:0/2644866157 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6240b1f50 con 0x7fd62409b3e0 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62ad40640 1 --2- 172.21.15.46:0/2644866157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409b3e0 0x7fd6240b3140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd61c0097a0 tx=0x7fd61c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d878ad29d381faad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62953d640 1 -- 172.21.15.46:0/2644866157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c035020 con 0x7fd62409b3e0 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62953d640 1 -- 172.21.15.46:0/2644866157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd61c031830 con 0x7fd62409b3e0 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62953d640 1 -- 172.21.15.46:0/2644866157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c031c00 con 0x7fd62409b3e0 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62c7a3640 1 -- 172.21.15.46:0/2644866157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409b3e0 msgr2=0x7fd6240b3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/2644866157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409b3e0 0x7fd6240b3140 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd61c0097a0 tx=0x7fd61c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62c7a3640 1 -- 172.21.15.46:0/2644866157 shutdown_connections 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/2644866157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409b3e0 0x7fd6240b3140 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.970+0000 7fd62c7a3640 1 -- 172.21.15.46:0/2644866157 >> 172.21.15.46:0/2644866157 conn(0x7fd6240b3e50 msgr2=0x7fd6240b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.971+0000 7fd62c7a3640 1 -- 172.21.15.46:0/2644866157 shutdown_connections 2024-04-15T15:16:05.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.971+0000 7fd62c7a3640 1 -- 172.21.15.46:0/2644866157 wait complete. 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 Processor -- start 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 -- start start 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd62409b3e0 0x7fd62415a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 0x7fd62415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240d0940 0x7fd62415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd62415ca30 con 0x7fd62409ba10 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd6240b2510 con 0x7fd6240d0940 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd62c7a3640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd62415cbb0 con 0x7fd62409b3e0 2024-04-15T15:16:05.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.972+0000 7fd629d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 0x7fd62415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 0x7fd62415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42368/0 (socket says 172.21.15.46:42368) 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 -- 172.21.15.46:0/3030240724 learned_addr learned my addr 172.21.15.46:0/3030240724 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd62a53f640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd62409b3e0 0x7fd62415a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd62ad40640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240d0940 0x7fd62415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 -- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd62409b3e0 msgr2=0x7fd62415a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd62409b3e0 0x7fd62415a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 -- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240d0940 msgr2=0x7fd62415b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240d0940 0x7fd62415b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 -- 172.21.15.46:0/3030240724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6240c9640 con 0x7fd62409ba10 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd62a53f640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd62409b3e0 0x7fd62415a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd62ad40640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240d0940 0x7fd62415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd629d3e640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 0x7fd62415b0a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd61800c400 tx=0x7fd61800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618017070 con 0x7fd62409ba10 2024-04-15T15:16:05.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6180052d0 con 0x7fd62409ba10 2024-04-15T15:16:05.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6240c9920 con 0x7fd62409ba10 2024-04-15T15:16:05.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618003830 con 0x7fd62409ba10 2024-04-15T15:16:05.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.973+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6240c9ee0 con 0x7fd62409ba10 2024-04-15T15:16:05.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.974+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6240b3b50 con 0x7fd62409ba10 2024-04-15T15:16:05.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.975+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd61801a070 con 0x7fd62409ba10 2024-04-15T15:16:05.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.975+0000 7fd622ffd640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd61006c8f0 0x7fd61006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.975+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd61808b1e0 con 0x7fd62409ba10 2024-04-15T15:16:05.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.975+0000 7fd62a53f640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd61006c8f0 0x7fd61006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.976+0000 7fd62a53f640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd61006c8f0 0x7fd61006edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd6140012e0 tx=0x7fd614005c30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:05.979+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6240b3b50 con 0x7fd62409ba10 2024-04-15T15:16:06.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.203+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd6240689e0 con 0x7fd62409ba10 2024-04-15T15:16:06.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.203+0000 7fd622ffd640 1 -- 172.21.15.46:0/3030240724 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd6180592d0 con 0x7fd62409ba10 2024-04-15T15:16:06.205 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:06.205 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd61006c8f0 msgr2=0x7fd61006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd61006c8f0 0x7fd61006edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd6140012e0 tx=0x7fd614005c30 comp rx=0 tx=0).stop 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 msgr2=0x7fd62415b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 0x7fd62415b0a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd61800c400 tx=0x7fd61800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 shutdown_connections 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd61006c8f0 0x7fd61006edb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd6240d0940 0x7fd62415b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd62409ba10 0x7fd62415b0a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 --2- 172.21.15.46:0/3030240724 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd62409b3e0 0x7fd62415a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 >> 172.21.15.46:0/3030240724 conn(0x7fd6240b3e50 msgr2=0x7fd6240cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 shutdown_connections 2024-04-15T15:16:06.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.205+0000 7fd62c7a3640 1 -- 172.21.15.46:0/3030240724 wait complete. 2024-04-15T15:16:06.218 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-15T15:16:06.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.320+0000 7fbe1362a640 1 Processor -- start 2024-04-15T15:16:06.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.320+0000 7fbe1362a640 1 -- start start 2024-04-15T15:16:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.321+0000 7fbe1362a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0d2360 0x7fbe0c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.321+0000 7fbe1362a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0c0b47a0 con 0x7fbe0c15eae0 2024-04-15T15:16:06.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.321+0000 7fbe1362a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0c0b4a80 con 0x7fbe0c15e700 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.321+0000 7fbe1362a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0c0b53b0 con 0x7fbe0c0d2360 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.321+0000 7fbe11bc7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe0c15eae0 0x7fbe0c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55698/0 (socket says 172.21.15.46:55698) 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.321+0000 7fbe11bc7640 1 -- 172.21.15.46:0/2613161513 learned_addr learned my addr 172.21.15.46:0/2613161513 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe10bc5640 1 --2- 172.21.15.46:0/2613161513 >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0d2360 0x7fbe0c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2398433845 0 0) 0x7fbe0c0b47a0 con 0x7fbe0c15eae0 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdf4003520 con 0x7fbe0c15eae0 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 448941741 0 0) 0x7fbdf4003520 con 0x7fbe0c15eae0 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0d2360 msgr2=0x7fbe0c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 --2- 172.21.15.46:0/2613161513 >> v2:172.21.15.78:3300/0 conn(0x7fbe0c0d2360 0x7fbe0c0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 >> v1:172.21.15.78:6789/0 conn(0x7fbe0c15e700 legacy=0x7fbe0c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c0b4d60 con 0x7fbe0c15eae0 2024-04-15T15:16:06.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fbdfc0019f0 con 0x7fbe0c15eae0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 >> v1:172.21.15.46:6789/0 conn(0x7fbe0c15eae0 legacy=0x7fbe0c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbdf40039b0 0x7fbdf4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdf40043d0 0x7fbdf40048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbdf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c0b53b0 con 0x7fbdf40050a0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf400bfe0 con 0x7fbdf40043d0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf400c1e0 con 0x7fbdf40039b0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe11bc7640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbdf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe113c6640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbdf40039b0 0x7fbdf4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe10bc5640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdf40043d0 0x7fbdf40048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe11bc7640 1 -- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbdf40039b0 msgr2=0x7fbdf4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe11bc7640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbdf40039b0 0x7fbdf4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe11bc7640 1 -- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdf40043d0 msgr2=0x7fbdf40048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe11bc7640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbdf40043d0 0x7fbdf40048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.322+0000 7fbe11bc7640 1 -- 172.21.15.46:0/2613161513 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf4003520 con 0x7fbdf40050a0 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.323+0000 7fbe11bc7640 1 -- 172.21.15.46:0/2613161513 reap_dead start 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.323+0000 7fbe11bc7640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbdf400b9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00a860 tx=0x7fbdf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b2a45d43c1c2ad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.323+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc00a480 con 0x7fbdf40050a0 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.323+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdfc034630 con 0x7fbdf40050a0 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.323+0000 7fbe0bfff640 1 -- 172.21.15.46:0/2613161513 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc034a00 con 0x7fbdf40050a0 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 -- 172.21.15.46:0/2613161513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 msgr2=0x7fbdf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbdf400b9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00a860 tx=0x7fbdf4005830 comp rx=0 tx=0).stop 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 -- 172.21.15.46:0/2613161513 shutdown_connections 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 --2- 172.21.15.46:0/2613161513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbdf400b9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 -- 172.21.15.46:0/2613161513 >> 172.21.15.46:0/2613161513 conn(0x7fbe0c08bf60 msgr2=0x7fbe0c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 -- 172.21.15.46:0/2613161513 shutdown_connections 2024-04-15T15:16:06.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.324+0000 7fbe1362a640 1 -- 172.21.15.46:0/2613161513 wait complete. 2024-04-15T15:16:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.325+0000 7fbe1362a640 1 Processor -- start 2024-04-15T15:16:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.325+0000 7fbe1362a640 1 -- start start 2024-04-15T15:16:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe1362a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbe0c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe1362a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0cc010 0x7fbe0c0b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe1362a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe0c0b87e0 0x7fbe0c0b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe1362a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c0d1d40 con 0x7fbdf40050a0 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe1362a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c0b4450 con 0x7fbe0c0cc010 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe1362a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0c0d1ec0 con 0x7fbe0c0b87e0 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbe0c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbe0c0cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42380/0 (socket says 172.21.15.46:42380) 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 -- 172.21.15.46:0/3578495822 learned_addr learned my addr 172.21.15.46:0/3578495822 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe11bc7640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe0c0b87e0 0x7fbe0c0b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe10bc5640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0cc010 0x7fbe0c0b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 -- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe0c0b87e0 msgr2=0x7fbe0c0b8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe0c0b87e0 0x7fbe0c0b8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 -- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0cc010 msgr2=0x7fbe0c0b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0cc010 0x7fbe0c0b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.326+0000 7fbe113c6640 1 -- 172.21.15.46:0/3578495822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c0cd130 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.327+0000 7fbe113c6640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbe0c0cde80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400f820 tx=0x7fbe0400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.327+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe040052e0 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.327+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe04018070 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.327+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400d970 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.327+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c0cc5b0 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.327+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c0ccb70 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.328+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe0c0689e0 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.328+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe0400db10 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.329+0000 7fbe09ffb640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf806c8f0 0x7fbdf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.329+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbe0408bcc0 con 0x7fbdf40050a0 2024-04-15T15:16:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.329+0000 7fbe10bc5640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf806c8f0 0x7fbdf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.329+0000 7fbe10bc5640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf806c8f0 0x7fbdf806edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c15cfe0 tx=0x7fbe00008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.333+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe04092080 con 0x7fbdf40050a0 2024-04-15T15:16:06.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.518+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe0c0ba500 con 0x7fbdf40050a0 2024-04-15T15:16:06.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.519+0000 7fbe09ffb640 1 -- 172.21.15.46:0/3578495822 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fbe040054a0 con 0x7fbdf40050a0 2024-04-15T15:16:06.521 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:06.522 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf806c8f0 msgr2=0x7fbdf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf806c8f0 0x7fbdf806edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c15cfe0 tx=0x7fbe00008040 comp rx=0 tx=0).stop 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 msgr2=0x7fbe0c0cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbe0c0cde80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400f820 tx=0x7fbe0400fce0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 shutdown_connections 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf806c8f0 0x7fbdf806edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe0c0b87e0 0x7fbe0c0b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe0c0cc010 0x7fbe0c0b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 --2- 172.21.15.46:0/3578495822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbdf40050a0 0x7fbe0c0cde80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 >> 172.21.15.46:0/3578495822 conn(0x7fbe0c08bf60 msgr2=0x7fbe0c0cfd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 shutdown_connections 2024-04-15T15:16:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.522+0000 7fbe1362a640 1 -- 172.21.15.46:0/3578495822 wait complete. 2024-04-15T15:16:06.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:06.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.643+0000 7f41621a8640 1 Processor -- start 2024-04-15T15:16:06.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f41621a8640 1 -- start start 2024-04-15T15:16:06.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f41621a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f415c15e9d0 0x7f415c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f41621a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f415c09af10 0x7f415c1683a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f41621a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415c09b320 con 0x7f415c15e5f0 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f41621a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f415c06dd30 con 0x7f415c15e9d0 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f41621a8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f415c06cc90 con 0x7f415c09af10 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f415affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f415c15e9d0 0x7f415c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f415b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f415c15e5f0 0x7f415c0d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55712/0 (socket says 172.21.15.46:55712) 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f415affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f415c15e9d0 0x7f415c09a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42396/0 (socket says 172.21.15.46:42396) 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.644+0000 7f415b7fe640 1 -- 172.21.15.46:0/4230680696 learned_addr learned my addr 172.21.15.46:0/4230680696 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.645+0000 7f415bfff640 1 --2- 172.21.15.46:0/4230680696 >> v2:172.21.15.78:3300/0 conn(0x7f415c09af10 0x7f415c1683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.645+0000 7f415affd640 1 -- 172.21.15.46:0/4230680696 >> v2:172.21.15.78:3300/0 conn(0x7f415c09af10 msgr2=0x7f415c1683a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.645+0000 7f415affd640 1 --2- 172.21.15.46:0/4230680696 >> v2:172.21.15.78:3300/0 conn(0x7f415c09af10 0x7f415c1683a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.645+0000 7f415affd640 1 -- 172.21.15.46:0/4230680696 >> v1:172.21.15.46:6789/0 conn(0x7f415c15e5f0 legacy=0x7f415c0d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.645+0000 7f415affd640 1 -- 172.21.15.46:0/4230680696 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c06cf70 con 0x7f415c15e9d0 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f415affd640 1 --2- 172.21.15.46:0/4230680696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c09a9d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f415c0cc9a0 tx=0x7f415002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c44a3ee4be1c7b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f415a7fc640 1 -- 172.21.15.46:0/4230680696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150035020 con 0x7f415c15e9d0 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f415a7fc640 1 -- 172.21.15.46:0/4230680696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4150031830 con 0x7f415c15e9d0 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f415a7fc640 1 -- 172.21.15.46:0/4230680696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150031c00 con 0x7f415c15e9d0 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 -- 172.21.15.46:0/4230680696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 msgr2=0x7f415c09a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 --2- 172.21.15.46:0/4230680696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c09a9d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f415c0cc9a0 tx=0x7f415002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 -- 172.21.15.46:0/4230680696 shutdown_connections 2024-04-15T15:16:06.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 --2- 172.21.15.46:0/4230680696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c09a9d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 --2- 172.21.15.46:0/4230680696 >> v2:172.21.15.78:3300/0 conn(0x7f415c09af10 0x7f415c1683a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 -- 172.21.15.46:0/4230680696 >> 172.21.15.46:0/4230680696 conn(0x7f415c0b3eb0 msgr2=0x7f415c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 -- 172.21.15.46:0/4230680696 shutdown_connections 2024-04-15T15:16:06.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.646+0000 7f41621a8640 1 -- 172.21.15.46:0/4230680696 wait complete. 2024-04-15T15:16:06.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.647+0000 7f41621a8640 1 Processor -- start 2024-04-15T15:16:06.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 -- start start 2024-04-15T15:16:06.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c09af10 0x7f415c15fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f415c15e5f0 0x7f415c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f415c06d810 con 0x7f415c15e9d0 2024-04-15T15:16:06.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f415c06d530 con 0x7f415c09af10 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f41621a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f415c0d1740 con 0x7f415c15e5f0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f415bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f415bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c161400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42412/0 (socket says 172.21.15.46:42412) 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.648+0000 7f415bfff640 1 -- 172.21.15.46:0/2969897165 learned_addr learned my addr 172.21.15.46:0/2969897165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415affd640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f415c15e5f0 0x7f415c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415b7fe640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c09af10 0x7f415c15fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415bfff640 1 -- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f415c15e5f0 msgr2=0x7f415c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415bfff640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f415c15e5f0 0x7f415c160ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415bfff640 1 -- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c09af10 msgr2=0x7f415c15fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415bfff640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c09af10 0x7f415c15fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415bfff640 1 -- 172.21.15.46:0/2969897165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c155b90 con 0x7f415c15e9d0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f415bfff640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c161400 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4154007e60 tx=0x7f415400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154014420 con 0x7f415c15e9d0 2024-04-15T15:16:06.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4154019070 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154014820 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415c0c8530 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.649+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415c0c8af0 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.650+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f415c0690f0 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.651+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41540149c0 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.651+0000 7f4143fff640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f414806c8f0 0x7f414806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.651+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f415408c6f0 con 0x7f415c15e9d0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.651+0000 7f415b7fe640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f414806c8f0 0x7f414806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.652+0000 7f415b7fe640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f414806c8f0 0x7f414806edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f414c000f60 tx=0x7f414c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.656+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f415405a7e0 con 0x7f415c15e9d0 2024-04-15T15:16:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.841+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f415c15f090 con 0x7f415c15e9d0 2024-04-15T15:16:06.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.842+0000 7f4143fff640 1 -- 172.21.15.46:0/2969897165 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f415405a0e0 con 0x7f415c15e9d0 2024-04-15T15:16:06.844 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f414806c8f0 msgr2=0x7f414806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f414806c8f0 0x7f414806edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f414c000f60 tx=0x7f414c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 msgr2=0x7f415c161400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c161400 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4154007e60 tx=0x7f415400e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 shutdown_connections 2024-04-15T15:16:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f414806c8f0 0x7f414806edb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f415c15e9d0 0x7f415c161400 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f415c15e5f0 0x7f415c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 --2- 172.21.15.46:0/2969897165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f415c09af10 0x7f415c15fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 >> 172.21.15.46:0/2969897165 conn(0x7f415c0b3eb0 msgr2=0x7f415c0b79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 shutdown_connections 2024-04-15T15:16:06.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.844+0000 7f41621a8640 1 -- 172.21.15.46:0/2969897165 wait complete. 2024-04-15T15:16:06.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:06.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.963+0000 7faf5d589640 1 Processor -- start 2024-04-15T15:16:06.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.963+0000 7faf5d589640 1 -- start start 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf5d589640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5815c8d0 0x7faf580cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf5d589640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7faf5809b650 0x7faf5809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf5d589640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf5809af10 0x7faf580d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf5d589640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7faf580b5b00 con 0x7faf5809af10 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf5d589640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7faf580d0350 con 0x7faf5809b650 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf5d589640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7faf580ce100 con 0x7faf5815c8d0 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf5809af10 0x7faf580d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faf5809af10 0x7faf580d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42414/0 (socket says 172.21.15.46:42414) 2024-04-15T15:16:06.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 -- 172.21.15.46:0/1448667059 learned_addr learned my addr 172.21.15.46:0/1448667059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf56ffd640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5815c8d0 0x7faf580cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf4ffff640 1 --2- 172.21.15.46:0/1448667059 >> v2:172.21.15.78:3300/0 conn(0x7faf5809b650 0x7faf5809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 -- 172.21.15.46:0/1448667059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5815c8d0 msgr2=0x7faf580cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5815c8d0 0x7faf580cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 -- 172.21.15.46:0/1448667059 >> v2:172.21.15.78:3300/0 conn(0x7faf5809b650 msgr2=0x7faf5809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 --2- 172.21.15.46:0/1448667059 >> v2:172.21.15.78:3300/0 conn(0x7faf5809b650 0x7faf5809a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf577fe640 1 -- 172.21.15.46:0/1448667059 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf580b54b0 con 0x7faf5809af10 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf56ffd640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5815c8d0 0x7faf580cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:06.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.964+0000 7faf4ffff640 1 --2- 172.21.15.46:0/1448667059 >> v2:172.21.15.78:3300/0 conn(0x7faf5809b650 0x7faf5809a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.965+0000 7faf577fe640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf5809af10 0x7faf580d0540 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf580687d0 tx=0x7faf5002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b47c3c83c1a37c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.965+0000 7faf567fc640 1 -- 172.21.15.46:0/1448667059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5002c9e0 con 0x7faf5809af10 2024-04-15T15:16:06.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.965+0000 7faf567fc640 1 -- 172.21.15.46:0/1448667059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf5002cb80 con 0x7faf5809af10 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.965+0000 7faf567fc640 1 -- 172.21.15.46:0/1448667059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf500343f0 con 0x7faf5809af10 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 -- 172.21.15.46:0/1448667059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf5809af10 msgr2=0x7faf580d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf5809af10 0x7faf580d0540 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faf580687d0 tx=0x7faf5002be40 comp rx=0 tx=0).stop 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 -- 172.21.15.46:0/1448667059 shutdown_connections 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf5809af10 0x7faf580d0540 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 --2- 172.21.15.46:0/1448667059 >> v2:172.21.15.78:3300/0 conn(0x7faf5809b650 0x7faf5809a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 --2- 172.21.15.46:0/1448667059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5815c8d0 0x7faf580cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 -- 172.21.15.46:0/1448667059 >> 172.21.15.46:0/1448667059 conn(0x7faf580b3fb0 msgr2=0x7faf580b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 -- 172.21.15.46:0/1448667059 shutdown_connections 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.966+0000 7faf5d589640 1 -- 172.21.15.46:0/1448667059 wait complete. 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.967+0000 7faf5d589640 1 Processor -- start 2024-04-15T15:16:06.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.967+0000 7faf5d589640 1 -- start start 2024-04-15T15:16:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5809af10 0x7faf58151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf5809b650 0x7faf580c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 0x7faf5809e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faf580d0c00 con 0x7faf580c8140 2024-04-15T15:16:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7faf580d0a80 con 0x7faf5809b650 2024-04-15T15:16:06.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7faf580d0d80 con 0x7faf5809af10 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 0x7faf5809e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 0x7faf5809e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42430/0 (socket says 172.21.15.46:42430) 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 -- 172.21.15.46:0/2065736248 learned_addr learned my addr 172.21.15.46:0/2065736248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf4ffff640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf5809b650 0x7faf580c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf56ffd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5809af10 0x7faf58151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 -- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5809af10 msgr2=0x7faf58151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5809af10 0x7faf58151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 -- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf5809b650 msgr2=0x7faf580c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf5809b650 0x7faf580c7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 -- 172.21.15.46:0/2065736248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5809c4d0 con 0x7faf580c8140 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf577fe640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 0x7faf5809e9e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf50036040 tx=0x7faf50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50043020 con 0x7faf580c8140 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf5003e450 con 0x7faf580c8140 2024-04-15T15:16:06.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5003e820 con 0x7faf580c8140 2024-04-15T15:16:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.968+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5809c7b0 con 0x7faf580c8140 2024-04-15T15:16:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.969+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5809cd70 con 0x7faf580c8140 2024-04-15T15:16:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.970+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf580690f0 con 0x7faf580c8140 2024-04-15T15:16:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.970+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf50046020 con 0x7faf580c8140 2024-04-15T15:16:06.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.970+0000 7faf4f7fe640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faf3806c8f0 0x7faf3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.971+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faf500ae680 con 0x7faf580c8140 2024-04-15T15:16:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.971+0000 7faf56ffd640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faf3806c8f0 0x7faf3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.971+0000 7faf56ffd640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faf3806c8f0 0x7faf3806edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faf4800aa40 tx=0x7faf48008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:06.975+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf5004d030 con 0x7faf580c8140 2024-04-15T15:16:07.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.159+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faf580cecb0 con 0x7faf580c8140 2024-04-15T15:16:07.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.160+0000 7faf4f7fe640 1 -- 172.21.15.46:0/2065736248 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7faf5007c1b0 con 0x7faf580c8140 2024-04-15T15:16:07.161 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faf3806c8f0 msgr2=0x7faf3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faf3806c8f0 0x7faf3806edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faf4800aa40 tx=0x7faf48008040 comp rx=0 tx=0).stop 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 msgr2=0x7faf5809e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 0x7faf5809e9e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faf50036040 tx=0x7faf50004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 shutdown_connections 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faf3806c8f0 0x7faf3806edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf580c8140 0x7faf5809e9e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faf5809b650 0x7faf580c7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 --2- 172.21.15.46:0/2065736248 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7faf5809af10 0x7faf58151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 >> 172.21.15.46:0/2065736248 conn(0x7faf580b3fb0 msgr2=0x7faf580b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 shutdown_connections 2024-04-15T15:16:07.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.162+0000 7faf5d589640 1 -- 172.21.15.46:0/2065736248 wait complete. 2024-04-15T15:16:07.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:07.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.281+0000 7fafe024e640 1 Processor -- start 2024-04-15T15:16:07.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafe024e640 1 -- start start 2024-04-15T15:16:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafe024e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafd80b0560 0x7fafd806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafe024e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fafd80b09a0 0x7fafd806bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafe024e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd80cbd10 con 0x7fafd815eb00 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafe024e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fafd815eee0 con 0x7fafd80b0560 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafe024e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fafd80cbff0 con 0x7fafd80b09a0 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafdd7e9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafd80b0560 0x7fafd806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafdd7e9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fafd80b0560 0x7fafd806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42446/0 (socket says 172.21.15.46:42446) 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafddfea640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fafd815eb00 0x7fafd80ae2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55716/0 (socket says 172.21.15.46:55716) 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.282+0000 7fafdd7e9640 1 -- 172.21.15.46:0/1822975039 learned_addr learned my addr 172.21.15.46:0/1822975039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.283+0000 7fafde7eb640 1 --2- 172.21.15.46:0/1822975039 >> v2:172.21.15.78:3300/0 conn(0x7fafd80b09a0 0x7fafd806bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.283+0000 7fafdd7e9640 1 -- 172.21.15.46:0/1822975039 >> v2:172.21.15.78:3300/0 conn(0x7fafd80b09a0 msgr2=0x7fafd806bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.283+0000 7fafdd7e9640 1 --2- 172.21.15.46:0/1822975039 >> v2:172.21.15.78:3300/0 conn(0x7fafd80b09a0 0x7fafd806bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.283+0000 7fafdd7e9640 1 -- 172.21.15.46:0/1822975039 >> v1:172.21.15.46:6789/0 conn(0x7fafd815eb00 legacy=0x7fafd80ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.283+0000 7fafdd7e9640 1 -- 172.21.15.46:0/1822975039 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd80cc2d0 con 0x7fafd80b0560 2024-04-15T15:16:07.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.283+0000 7fafdcfe8640 1 -- 172.21.15.46:0/1822975039 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 144600358 0 0) 0x7fafd80cbd10 con 0x7fafd815eb00 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafdd7e9640 1 --2- 172.21.15.46:0/1822975039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b0560 0x7fafd806b9e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fafd8072e60 tx=0x7fafcc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac51ab8b9a66b6c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafdcfe8640 1 -- 172.21.15.46:0/1822975039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc034070 con 0x7fafd80b0560 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafdcfe8640 1 -- 172.21.15.46:0/1822975039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafcc030860 con 0x7fafd80b0560 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafdcfe8640 1 -- 172.21.15.46:0/1822975039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc030c30 con 0x7fafd80b0560 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 -- 172.21.15.46:0/1822975039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b0560 msgr2=0x7fafd806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 --2- 172.21.15.46:0/1822975039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b0560 0x7fafd806b9e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fafd8072e60 tx=0x7fafcc02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 -- 172.21.15.46:0/1822975039 shutdown_connections 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 --2- 172.21.15.46:0/1822975039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b0560 0x7fafd806b9e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 --2- 172.21.15.46:0/1822975039 >> v2:172.21.15.78:3300/0 conn(0x7fafd80b09a0 0x7fafd806bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 -- 172.21.15.46:0/1822975039 >> 172.21.15.46:0/1822975039 conn(0x7fafd809a9d0 msgr2=0x7fafd80d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 -- 172.21.15.46:0/1822975039 shutdown_connections 2024-04-15T15:16:07.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.284+0000 7fafe024e640 1 -- 172.21.15.46:0/1822975039 wait complete. 2024-04-15T15:16:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 Processor -- start 2024-04-15T15:16:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 -- start start 2024-04-15T15:16:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fafd80b0560 0x7fafd80b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 0x7fafd80b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafd815eb00 0x7fafd80cac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fafd80cec60 con 0x7fafd80b09a0 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fafd80cc890 con 0x7fafd815eb00 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafe024e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fafd80cede0 con 0x7fafd80b0560 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.286+0000 7fafdd7e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 0x7fafd80b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 0x7fafd80b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42460/0 (socket says 172.21.15.46:42460) 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 -- 172.21.15.46:0/435441013 learned_addr learned my addr 172.21.15.46:0/435441013 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafddfea640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fafd80b0560 0x7fafd80b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafde7eb640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafd815eb00 0x7fafd80cac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 -- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fafd80b0560 msgr2=0x7fafd80b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fafd80b0560 0x7fafd80b6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 -- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafd815eb00 msgr2=0x7fafd80cac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafd815eb00 0x7fafd80cac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 -- 172.21.15.46:0/435441013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd80b4a90 con 0x7fafd80b09a0 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafddfea640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fafd80b0560 0x7fafd80b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafdd7e9640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 0x7fafd80b4190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fafcc02b9e0 tx=0x7fafcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc034040 con 0x7fafd80b09a0 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafcc0433f0 con 0x7fafd80b09a0 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0437e0 con 0x7fafd80b09a0 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd80b4d70 con 0x7fafd80b09a0 2024-04-15T15:16:07.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.287+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd80b5330 con 0x7fafd80b09a0 2024-04-15T15:16:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.288+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd80690f0 con 0x7fafd80b09a0 2024-04-15T15:16:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.289+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fafcc037070 con 0x7fafd80b09a0 2024-04-15T15:16:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.289+0000 7fafd67fc640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fafc406c920 0x7fafc406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.289+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fafcc0ae1a0 con 0x7fafd80b09a0 2024-04-15T15:16:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.289+0000 7fafddfea640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fafc406c920 0x7fafc406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.290+0000 7fafddfea640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fafc406c920 0x7fafc406ede0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fafd0000f60 tx=0x7fafd0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.293+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fafcc03e070 con 0x7fafd80b09a0 2024-04-15T15:16:07.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.479+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fafd80cf420 con 0x7fafd80b09a0 2024-04-15T15:16:07.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.480+0000 7fafd67fc640 1 -- 172.21.15.46:0/435441013 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fafcc07c3b0 con 0x7fafd80b09a0 2024-04-15T15:16:07.482 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fafc406c920 msgr2=0x7fafc406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fafc406c920 0x7fafc406ede0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fafd0000f60 tx=0x7fafd0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 msgr2=0x7fafd80b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 0x7fafd80b4190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fafcc02b9e0 tx=0x7fafcc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 shutdown_connections 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fafc406c920 0x7fafc406ede0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fafd815eb00 0x7fafd80cac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fafd80b09a0 0x7fafd80b4190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 --2- 172.21.15.46:0/435441013 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fafd80b0560 0x7fafd80b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 >> 172.21.15.46:0/435441013 conn(0x7fafd809a9d0 msgr2=0x7fafd8070250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 shutdown_connections 2024-04-15T15:16:07.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.482+0000 7fafe024e640 1 -- 172.21.15.46:0/435441013 wait complete. 2024-04-15T15:16:07.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.600+0000 7f3721ce4640 1 Processor -- start 2024-04-15T15:16:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.600+0000 7f3721ce4640 1 -- start start 2024-04-15T15:16:07.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f3721ce4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f371c0af5e0 0x7f371c0ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f3721ce4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f371c06ed20 con 0x7f371c06e560 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f3721ce4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f371c06f000 con 0x7f371c0b00d0 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f3721ce4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f371c0b6140 con 0x7f371c0af5e0 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f371b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f371c0af5e0 0x7f371c0ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f371affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f371c06e560 0x7f371c0af9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55732/0 (socket says 172.21.15.46:55732) 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f371affd640 1 -- 172.21.15.46:0/1105741390 learned_addr learned my addr 172.21.15.46:0/1105741390 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.601+0000 7f371b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f371c0af5e0 0x7f371c0ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42466/0 (socket says 172.21.15.46:42466) 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.602+0000 7f371b7fe640 1 -- 172.21.15.46:0/1105741390 >> v1:172.21.15.78:6789/0 conn(0x7f371c0b00d0 legacy=0x7f371c1682c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.602+0000 7f371b7fe640 1 -- 172.21.15.46:0/1105741390 >> v1:172.21.15.46:6789/0 conn(0x7f371c06e560 legacy=0x7f371c0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.602+0000 7f371b7fe640 1 -- 172.21.15.46:0/1105741390 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c06f2e0 con 0x7f371c0af5e0 2024-04-15T15:16:07.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.602+0000 7f371a7fc640 1 -- 172.21.15.46:0/1105741390 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410617156 0 0) 0x7f371c06ed20 con 0x7f371c06e560 2024-04-15T15:16:07.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f371b7fe640 1 --2- 172.21.15.46:0/1105741390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0af5e0 0x7f371c0ae8a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f370c0097a0 tx=0x7f370c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb68a8acbac4cfb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f371a7fc640 1 -- 172.21.15.46:0/1105741390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c034070 con 0x7f371c0af5e0 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f371a7fc640 1 -- 172.21.15.46:0/1105741390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f370c030610 con 0x7f371c0af5e0 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f371a7fc640 1 -- 172.21.15.46:0/1105741390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370c0309e0 con 0x7f371c0af5e0 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 -- 172.21.15.46:0/1105741390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0af5e0 msgr2=0x7f371c0ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 --2- 172.21.15.46:0/1105741390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0af5e0 0x7f371c0ae8a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f370c0097a0 tx=0x7f370c02b960 comp rx=0 tx=0).stop 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 -- 172.21.15.46:0/1105741390 shutdown_connections 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 --2- 172.21.15.46:0/1105741390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0af5e0 0x7f371c0ae8a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 -- 172.21.15.46:0/1105741390 >> 172.21.15.46:0/1105741390 conn(0x7f371c06b9e0 msgr2=0x7f371c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 -- 172.21.15.46:0/1105741390 shutdown_connections 2024-04-15T15:16:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.603+0000 7f3721ce4640 1 -- 172.21.15.46:0/1105741390 wait complete. 2024-04-15T15:16:07.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.604+0000 7f3721ce4640 1 Processor -- start 2024-04-15T15:16:07.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 -- start start 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f371c06e560 0x7f371c145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f371c0af5e0 0x7f371c1f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 0x7f371c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f371c06e9d0 con 0x7f371c0b00d0 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f371c06f8a0 con 0x7f371c0af5e0 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f3721ce4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f371c15ea40 con 0x7f371c06e560 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f371bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 0x7f371c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f371bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 0x7f371c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42478/0 (socket says 172.21.15.46:42478) 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.605+0000 7f371bfff640 1 -- 172.21.15.46:0/2383330524 learned_addr learned my addr 172.21.15.46:0/2383330524 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371b7fe640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f371c06e560 0x7f371c145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371affd640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f371c0af5e0 0x7f371c1f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371bfff640 1 -- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f371c06e560 msgr2=0x7f371c145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371bfff640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f371c06e560 0x7f371c145720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371bfff640 1 -- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f371c0af5e0 msgr2=0x7f371c1f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371bfff640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f371c0af5e0 0x7f371c1f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371bfff640 1 -- 172.21.15.46:0/2383330524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c1f4f90 con 0x7f371c0b00d0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371affd640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f371c0af5e0 0x7f371c1f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371b7fe640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f371c06e560 0x7f371c145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f371bfff640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 0x7f371c15ab70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f371400c6b0 tx=0x7f371400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714003b70 con 0x7f371c0b00d0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3714003d30 con 0x7f371c0b00d0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c1f5730 con 0x7f371c0b00d0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714005310 con 0x7f371c0b00d0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.606+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1f5cf0 con 0x7f371c0b00d0 2024-04-15T15:16:07.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.607+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c0690f0 con 0x7f371c0b00d0 2024-04-15T15:16:07.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.608+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f37140054b0 con 0x7f371c0b00d0 2024-04-15T15:16:07.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.608+0000 7f3703fff640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f370806c8f0 0x7f370806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.608+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f371408bc60 con 0x7f371c0b00d0 2024-04-15T15:16:07.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.608+0000 7f371b7fe640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f370806c8f0 0x7f370806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.609+0000 7f371b7fe640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f370806c8f0 0x7f370806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f370c036000 tx=0x7f370c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.612+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f371c1f4f90 con 0x7f371c0b00d0 2024-04-15T15:16:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.796+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f371c15b0b0 con 0x7f371c0b00d0 2024-04-15T15:16:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.796+0000 7f3703fff640 1 -- 172.21.15.46:0/2383330524 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f3714059de0 con 0x7f371c0b00d0 2024-04-15T15:16:07.798 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f370806c8f0 msgr2=0x7f370806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f370806c8f0 0x7f370806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f370c036000 tx=0x7f370c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 msgr2=0x7f371c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 0x7f371c15ab70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f371400c6b0 tx=0x7f371400cb70 comp rx=0 tx=0).stop 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 shutdown_connections 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f370806c8f0 0x7f370806edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f371c0b00d0 0x7f371c15ab70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f371c0af5e0 0x7f371c1f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 --2- 172.21.15.46:0/2383330524 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f371c06e560 0x7f371c145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.798+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 >> 172.21.15.46:0/2383330524 conn(0x7f371c06b9e0 msgr2=0x7f371c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.799+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 shutdown_connections 2024-04-15T15:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.799+0000 7f3721ce4640 1 -- 172.21.15.46:0/2383330524 wait complete. 2024-04-15T15:16:07.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:07.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.915+0000 7f0690fc8640 1 Processor -- start 2024-04-15T15:16:07.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.915+0000 7f0690fc8640 1 -- start start 2024-04-15T15:16:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f0690fc8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c15e7a0 0x7f068c0d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f0690fc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f068c0d1870 0x7f068c162a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f0690fc8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c0d0530 con 0x7f068c0d13f0 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f0690fc8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f068c0cf490 con 0x7f068c0d1870 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f0690fc8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f068c0cf770 con 0x7f068c15e7a0 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f068c0d1870 0x7f068c162a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f068c0d1870 0x7f068c162a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42486/0 (socket says 172.21.15.46:42486) 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 -- 172.21.15.46:0/1559715083 learned_addr learned my addr 172.21.15.46:0/1559715083 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068a59c640 1 --2- 172.21.15.46:0/1559715083 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c15e7a0 0x7f068c0d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 -- 172.21.15.46:0/1559715083 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c15e7a0 msgr2=0x7f068c0d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 --2- 172.21.15.46:0/1559715083 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c15e7a0 0x7f068c0d0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 -- 172.21.15.46:0/1559715083 >> v1:172.21.15.78:6789/0 conn(0x7f068c0d13f0 legacy=0x7f068c162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.916+0000 7f068ad9d640 1 -- 172.21.15.46:0/1559715083 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c0cfa50 con 0x7f068c0d1870 2024-04-15T15:16:07.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.917+0000 7f068ad9d640 1 --2- 172.21.15.46:0/1559715083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0d1870 0x7f068c162a80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0684009e50 tx=0x7f068402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4acb061f82fd86f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.917+0000 7f0689d9b640 1 -- 172.21.15.46:0/1559715083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068402ca70 con 0x7f068c0d1870 2024-04-15T15:16:07.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.917+0000 7f0689d9b640 1 -- 172.21.15.46:0/1559715083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f068402cc10 con 0x7f068c0d1870 2024-04-15T15:16:07.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.917+0000 7f0689d9b640 1 -- 172.21.15.46:0/1559715083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06840343f0 con 0x7f068c0d1870 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 -- 172.21.15.46:0/1559715083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0d1870 msgr2=0x7f068c162a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1559715083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0d1870 0x7f068c162a80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0684009e50 tx=0x7f068402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 -- 172.21.15.46:0/1559715083 shutdown_connections 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1559715083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c0d1870 0x7f068c162a80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1559715083 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c15e7a0 0x7f068c0d0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 -- 172.21.15.46:0/1559715083 >> 172.21.15.46:0/1559715083 conn(0x7f068c08bf60 msgr2=0x7f068c0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 -- 172.21.15.46:0/1559715083 shutdown_connections 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.918+0000 7f0690fc8640 1 -- 172.21.15.46:0/1559715083 wait complete. 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.919+0000 7f0690fc8640 1 Processor -- start 2024-04-15T15:16:07.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.919+0000 7f0690fc8640 1 -- start start 2024-04-15T15:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0690fc8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c0d13f0 0x7f068c0cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0690fc8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0d1870 0x7f068c0c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0690fc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 0x7f068c15c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0690fc8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f068c06db20 con 0x7f068c15e7a0 2024-04-15T15:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0690fc8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f068c0cf180 con 0x7f068c0d1870 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0690fc8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f068c15eb80 con 0x7f068c0d13f0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 0x7f068c15c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 0x7f068c15c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42488/0 (socket says 172.21.15.46:42488) 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 -- 172.21.15.46:0/1097384444 learned_addr learned my addr 172.21.15.46:0/1097384444 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068a59c640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c0d13f0 0x7f068c0cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f0683fff640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0d1870 0x7f068c0c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 -- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c0d13f0 msgr2=0x7f068c0cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c0d13f0 0x7f068c0cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 -- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0d1870 msgr2=0x7f068c0c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0d1870 0x7f068c0c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 -- 172.21.15.46:0/1097384444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c0c8fd0 con 0x7f068c15e7a0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.920+0000 7f068ad9d640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 0x7f068c15c690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0684036040 tx=0x7f0684004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.921+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684043020 con 0x7f068c15e7a0 2024-04-15T15:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.921+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f068403e450 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.921+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c0c9240 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.921+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684009510 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.921+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c0c9800 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.922+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c0689e0 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.922+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0684046020 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.923+0000 7f0682ffd640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f066c06c8f0 0x7f066c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.923+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f06840ae440 con 0x7f068c15e7a0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.923+0000 7f068a59c640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f066c06c8f0 0x7f066c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.923+0000 7f068a59c640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f066c06c8f0 0x7f066c06edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f068c15d340 tx=0x7f067c009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:07.927+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f068404d030 con 0x7f068c15e7a0 2024-04-15T15:16:08.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.150+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f068c0d1d60 con 0x7f068c15e7a0 2024-04-15T15:16:08.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.150+0000 7f0682ffd640 1 -- 172.21.15.46:0/1097384444 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f068407bf70 con 0x7f068c15e7a0 2024-04-15T15:16:08.152 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:08.152 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:08.153 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f066c06c8f0 msgr2=0x7f066c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f066c06c8f0 0x7f066c06edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f068c15d340 tx=0x7f067c009190 comp rx=0 tx=0).stop 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 msgr2=0x7f068c15c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 0x7f068c15c690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0684036040 tx=0x7f0684004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 shutdown_connections 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f066c06c8f0 0x7f066c06edb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f068c15e7a0 0x7f068c15c690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f068c0d1870 0x7f068c0c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 --2- 172.21.15.46:0/1097384444 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f068c0d13f0 0x7f068c0cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 >> 172.21.15.46:0/1097384444 conn(0x7f068c08bf60 msgr2=0x7f068c0b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 shutdown_connections 2024-04-15T15:16:08.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.152+0000 7f0690fc8640 1 -- 172.21.15.46:0/1097384444 wait complete. 2024-04-15T15:16:08.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:08.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.268+0000 7fb17351c640 1 Processor -- start 2024-04-15T15:16:08.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.268+0000 7fb17351c640 1 -- start start 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb17351c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb16c070940 0x7fb16c09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb17351c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb16c160310 0x7fb16c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb17351c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb16c0d05e0 con 0x7fb16c0d0da0 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb17351c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb16c071260 con 0x7fb16c070940 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb17351c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb16c15f720 con 0x7fb16c160310 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb1712b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb16c070940 0x7fb16c09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb171ab9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb16c0d0da0 0x7fb16c0d11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55738/0 (socket says 172.21.15.46:55738) 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb1712b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb16c070940 0x7fb16c09b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42502/0 (socket says 172.21.15.46:42502) 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb171ab9640 1 -- 172.21.15.46:0/3051203356 learned_addr learned my addr 172.21.15.46:0/3051203356 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb170ab7640 1 --2- 172.21.15.46:0/3051203356 >> v2:172.21.15.78:3300/0 conn(0x7fb16c160310 0x7fb16c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb1712b8640 1 -- 172.21.15.46:0/3051203356 >> v2:172.21.15.78:3300/0 conn(0x7fb16c160310 msgr2=0x7fb16c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb1712b8640 1 --2- 172.21.15.46:0/3051203356 >> v2:172.21.15.78:3300/0 conn(0x7fb16c160310 0x7fb16c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb1712b8640 1 -- 172.21.15.46:0/3051203356 >> v1:172.21.15.46:6789/0 conn(0x7fb16c0d0da0 legacy=0x7fb16c0d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb16bfff640 1 -- 172.21.15.46:0/3051203356 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764400372 0 0) 0x7fb16c0d05e0 con 0x7fb16c0d0da0 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.269+0000 7fb1712b8640 1 -- 172.21.15.46:0/3051203356 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c15fa00 con 0x7fb16c070940 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.270+0000 7fb1712b8640 1 --2- 172.21.15.46:0/3051203356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c070940 0x7fb16c09b880 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0734f0 tx=0x7fb15c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522ec6eab150da2e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.270+0000 7fb16bfff640 1 -- 172.21.15.46:0/3051203356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c039070 con 0x7fb16c070940 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.270+0000 7fb16bfff640 1 -- 172.21.15.46:0/3051203356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb15c02cc80 con 0x7fb16c070940 2024-04-15T15:16:08.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.270+0000 7fb16bfff640 1 -- 172.21.15.46:0/3051203356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c034660 con 0x7fb16c070940 2024-04-15T15:16:08.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 -- 172.21.15.46:0/3051203356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c070940 msgr2=0x7fb16c09b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 --2- 172.21.15.46:0/3051203356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c070940 0x7fb16c09b880 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0734f0 tx=0x7fb15c02bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 -- 172.21.15.46:0/3051203356 shutdown_connections 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 --2- 172.21.15.46:0/3051203356 >> v2:172.21.15.78:3300/0 conn(0x7fb16c160310 0x7fb16c070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 --2- 172.21.15.46:0/3051203356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c070940 0x7fb16c09b880 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 -- 172.21.15.46:0/3051203356 >> 172.21.15.46:0/3051203356 conn(0x7fb16c0b3f40 msgr2=0x7fb16c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 -- 172.21.15.46:0/3051203356 shutdown_connections 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.271+0000 7fb17351c640 1 -- 172.21.15.46:0/3051203356 wait complete. 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.272+0000 7fb17351c640 1 Processor -- start 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.272+0000 7fb17351c640 1 -- start start 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb16c070940 0x7fb16c1f8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb16c0d0da0 0x7fb16c06dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 0x7fb16c1f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c0d18e0 con 0x7fb16c160310 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c0d0880 con 0x7fb16c070940 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb16c160770 con 0x7fb16c0d0da0 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 0x7fb16c1f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 0x7fb16c1f39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42512/0 (socket says 172.21.15.46:42512) 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 -- 172.21.15.46:0/723473176 learned_addr learned my addr 172.21.15.46:0/723473176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb170ab7640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb16c0d0da0 0x7fb16c06dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb1712b8640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb16c070940 0x7fb16c1f8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 -- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb16c0d0da0 msgr2=0x7fb16c06dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb16c0d0da0 0x7fb16c06dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 -- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb16c070940 msgr2=0x7fb16c1f8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb16c070940 0x7fb16c1f8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 -- 172.21.15.46:0/723473176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c1f8a70 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb171ab9640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 0x7fb16c1f39a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb16400c440 tx=0x7fb16400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1640074e0 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1640047e0 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c0b6b10 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.273+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164004bd0 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.274+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c0b7090 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.275+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb16c0689e0 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.275+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb164011070 con 0x7fb16c160310 2024-04-15T15:16:08.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.275+0000 7fb169ffb640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb15806c8f0 0x7fb15806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.276+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb16408b8c0 con 0x7fb16c160310 2024-04-15T15:16:08.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.276+0000 7fb1712b8640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb15806c8f0 0x7fb15806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.276+0000 7fb1712b8640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb15806c8f0 0x7fb15806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb15c036000 tx=0x7fb15c030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.280+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb16c1f8a70 con 0x7fb16c160310 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.464+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb16c070720 con 0x7fb16c160310 2024-04-15T15:16:08.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.466+0000 7fb169ffb640 1 -- 172.21.15.46:0/723473176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fb164059360 con 0x7fb16c160310 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:08.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.467+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb15806c8f0 msgr2=0x7fb15806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb15806c8f0 0x7fb15806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb15c036000 tx=0x7fb15c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 msgr2=0x7fb16c1f39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 0x7fb16c1f39a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb16400c440 tx=0x7fb16400c900 comp rx=0 tx=0).stop 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 shutdown_connections 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb15806c8f0 0x7fb15806edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16c160310 0x7fb16c1f39a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb16c0d0da0 0x7fb16c06dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 --2- 172.21.15.46:0/723473176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb16c070940 0x7fb16c1f8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 >> 172.21.15.46:0/723473176 conn(0x7fb16c0b3f40 msgr2=0x7fb16c0b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 shutdown_connections 2024-04-15T15:16:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.468+0000 7fb17351c640 1 -- 172.21.15.46:0/723473176 wait complete. 2024-04-15T15:16:08.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:08.482 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:08.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 Processor -- start 2024-04-15T15:16:08.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 -- start start 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2ecc070940 0x7f2ecc09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ecc0d0da0 0x7f2ecc0d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ecc0d05e0 con 0x7f2ecc160310 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2ecc15f720 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.588+0000 7f2ed21e1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2ecc15fa00 con 0x7f2ecc070940 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ecc0d0da0 0x7f2ecc0d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2ecc160310 0x7f2ecc070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55740/0 (socket says 172.21.15.46:55740) 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2ecc0d0da0 0x7f2ecc0d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42516/0 (socket says 172.21.15.46:42516) 2024-04-15T15:16:08.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecaffd640 1 -- 172.21.15.46:0/1560035592 learned_addr learned my addr 172.21.15.46:0/1560035592 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecb7fe640 1 --2- 172.21.15.46:0/1560035592 >> v2:172.21.15.78:3300/0 conn(0x7f2ecc070940 0x7f2ecc09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecbfff640 1 -- 172.21.15.46:0/1560035592 >> v2:172.21.15.78:3300/0 conn(0x7f2ecc070940 msgr2=0x7f2ecc09b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecbfff640 1 --2- 172.21.15.46:0/1560035592 >> v2:172.21.15.78:3300/0 conn(0x7f2ecc070940 0x7f2ecc09b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecbfff640 1 -- 172.21.15.46:0/1560035592 >> v1:172.21.15.46:6789/0 conn(0x7f2ecc160310 legacy=0x7f2ecc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2eca7fc640 1 -- 172.21.15.46:0/1560035592 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4009469676 0 0) 0x7f2ecc0d05e0 con 0x7f2ecc160310 2024-04-15T15:16:08.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.589+0000 7f2ecbfff640 1 -- 172.21.15.46:0/1560035592 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc15fce0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ecbfff640 1 --2- 172.21.15.46:0/1560035592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0d11f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0741c0 tx=0x7f2ec402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58828a546287605 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2eca7fc640 1 -- 172.21.15.46:0/1560035592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4039070 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2eca7fc640 1 -- 172.21.15.46:0/1560035592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec402cc80 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2eca7fc640 1 -- 172.21.15.46:0/1560035592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec4034660 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ed21e1640 1 -- 172.21.15.46:0/1560035592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 msgr2=0x7f2ecc0d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/1560035592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0d11f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0741c0 tx=0x7f2ec402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ed21e1640 1 -- 172.21.15.46:0/1560035592 shutdown_connections 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/1560035592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0d11f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/1560035592 >> v2:172.21.15.78:3300/0 conn(0x7f2ecc070940 0x7f2ecc09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.590+0000 7f2ed21e1640 1 -- 172.21.15.46:0/1560035592 >> 172.21.15.46:0/1560035592 conn(0x7f2ecc0b3f40 msgr2=0x7f2ecc0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.591+0000 7f2ed21e1640 1 -- 172.21.15.46:0/1560035592 shutdown_connections 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.591+0000 7f2ed21e1640 1 -- 172.21.15.46:0/1560035592 wait complete. 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.592+0000 7f2ed21e1640 1 Processor -- start 2024-04-15T15:16:08.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.592+0000 7f2ed21e1640 1 -- start start 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ed21e1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ecc070940 0x7f2ecc15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ed21e1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ed21e1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ecc160310 0x7f2ecc1f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ed21e1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc0d1730 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ed21e1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2ecc160770 con 0x7f2ecc160310 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ed21e1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2ecc0d18b0 con 0x7f2ecc070940 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0b7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42526/0 (socket says 172.21.15.46:42526) 2024-04-15T15:16:08.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 -- 172.21.15.46:0/3902452075 learned_addr learned my addr 172.21.15.46:0/3902452075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecb7fe640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ecc070940 0x7f2ecc15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecbfff640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ecc160310 0x7f2ecc1f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 -- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ecc070940 msgr2=0x7f2ecc15c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ecc070940 0x7f2ecc15c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 -- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ecc160310 msgr2=0x7f2ecc1f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ecc160310 0x7f2ecc1f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 -- 172.21.15.46:0/3902452075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ecc0b8190 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ecaffd640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0b7c50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c430 tx=0x7f2eb400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb40074e0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.593+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb40047e0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.594+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ecc0b5480 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.594+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4004bd0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.594+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ecc0b6680 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.595+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ecc0689e0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.595+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2eb4011070 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.596+0000 7f2ec3fff640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2eac06c8f0 0x7f2eac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.596+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2eb408b8c0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.596+0000 7f2ecb7fe640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2eac06c8f0 0x7f2eac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.596+0000 7f2ecb7fe640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2eac06c8f0 0x7f2eac06edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00aa20 tx=0x7f2ebc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.600+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2eb40593f0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.785+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ecc070720 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.786+0000 7f2ec3fff640 1 -- 172.21.15.46:0/3902452075 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f2eb405d6f0 con 0x7f2ecc0d0da0 2024-04-15T15:16:08.791 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2eac06c8f0 msgr2=0x7f2eac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2eac06c8f0 0x7f2eac06edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc00aa20 tx=0x7f2ebc008040 comp rx=0 tx=0).stop 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 msgr2=0x7f2ecc0b7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0b7c50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c430 tx=0x7f2eb400c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 shutdown_connections 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2eac06c8f0 0x7f2eac06edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2ecc160310 0x7f2ecc1f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ecc0d0da0 0x7f2ecc0b7c50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 --2- 172.21.15.46:0/3902452075 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2ecc070940 0x7f2ecc15c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 >> 172.21.15.46:0/3902452075 conn(0x7f2ecc0b3f40 msgr2=0x7f2ecc070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 shutdown_connections 2024-04-15T15:16:08.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.788+0000 7f2ed21e1640 1 -- 172.21.15.46:0/3902452075 wait complete. 2024-04-15T15:16:08.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.906+0000 7f9d70cb3640 1 Processor -- start 2024-04-15T15:16:08.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.906+0000 7f9d70cb3640 1 -- start start 2024-04-15T15:16:08.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.907+0000 7f9d70cb3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9d6c15eb00 0x7f9d6c0ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.907+0000 7f9d70cb3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0cbd10 con 0x7f9d6c06c0f0 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.907+0000 7f9d70cb3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0cbff0 con 0x7f9d6c0b0560 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.907+0000 7f9d70cb3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f9d6c15eee0 con 0x7f9d6c15eb00 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.907+0000 7f9d6ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d6c06c0f0 0x7f9d6c0ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55748/0 (socket says 172.21.15.46:55748) 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.907+0000 7f9d6ad9d640 1 -- 172.21.15.46:0/3222167022 learned_addr learned my addr 172.21.15.46:0/3222167022 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/3222167022 >> v2:172.21.15.78:3300/0 conn(0x7f9d6c15eb00 0x7f9d6c0ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914264290 0 0) 0x7f9d6c0cbd10 con 0x7f9d6c06c0f0 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d58003520 con 0x7f9d6c06c0f0 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3466727985 0 0) 0x7f9d58003520 con 0x7f9d6c06c0f0 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 >> v2:172.21.15.78:3300/0 conn(0x7f9d6c15eb00 msgr2=0x7f9d6c0ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 --2- 172.21.15.46:0/3222167022 >> v2:172.21.15.78:3300/0 conn(0x7f9d6c15eb00 0x7f9d6c0ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 >> v1:172.21.15.78:6789/0 conn(0x7f9d6c0b0560 legacy=0x7f9d6c06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c0cc2d0 con 0x7f9d6c06c0f0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f9d5c0019f0 con 0x7f9d6c06c0f0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 >> v1:172.21.15.46:6789/0 conn(0x7f9d6c06c0f0 legacy=0x7f9d6c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d580043b0 0x7f9d58004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d58005080 0x7f9d5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c15eee0 con 0x7f9d580039b0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5800bfe0 con 0x7f9d58005080 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5800c1e0 con 0x7f9d580043b0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6ad9d640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d58005080 0x7f9d5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d69d9b640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d580043b0 0x7f9d58004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 -- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d580043b0 msgr2=0x7f9d58004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d580043b0 0x7f9d58004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 -- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d58005080 msgr2=0x7f9d5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d58005080 0x7f9d5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6a59c640 1 -- 172.21.15.46:0/3222167022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d58003520 con 0x7f9d580039b0 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d6ad9d640 1 -- 172.21.15.46:0/3222167022 reap_dead start 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.908+0000 7f9d69d9b640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d580043b0 0x7f9d58004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.909+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d58003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009ac0 tx=0x7f9d58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d27ea9b5a99d032 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.909+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64039070 con 0x7f9d580039b0 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.909+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6402c6b0 con 0x7f9d580039b0 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.909+0000 7f9d6959a640 1 -- 172.21.15.46:0/3222167022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64009690 con 0x7f9d580039b0 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 -- 172.21.15.46:0/3222167022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 msgr2=0x7f9d58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d58003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009ac0 tx=0x7f9d58005830 comp rx=0 tx=0).stop 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 -- 172.21.15.46:0/3222167022 shutdown_connections 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/3222167022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d58003d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 -- 172.21.15.46:0/3222167022 >> 172.21.15.46:0/3222167022 conn(0x7f9d6c09a9d0 msgr2=0x7f9d6c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 -- 172.21.15.46:0/3222167022 shutdown_connections 2024-04-15T15:16:08.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.910+0000 7f9d70cb3640 1 -- 172.21.15.46:0/3222167022 wait complete. 2024-04-15T15:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.911+0000 7f9d70cb3640 1 Processor -- start 2024-04-15T15:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.911+0000 7f9d70cb3640 1 -- start start 2024-04-15T15:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d70cb3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d6c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d70cb3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d6c0c5080 0x7f9d6c1f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d70cb3640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d6c1509b0 0x7f9d6c0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d70cb3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0cec60 con 0x7f9d580039b0 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d70cb3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0cc890 con 0x7f9d6c0c5080 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d70cb3640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6c0cede0 con 0x7f9d6c1509b0 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d6c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d6c0b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42544/0 (socket says 172.21.15.46:42544) 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 -- 172.21.15.46:0/855405957 learned_addr learned my addr 172.21.15.46:0/855405957 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6ad9d640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d6c1509b0 0x7f9d6c0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d69d9b640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d6c0c5080 0x7f9d6c1f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 -- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d6c1509b0 msgr2=0x7f9d6c0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d6c1509b0 0x7f9d6c0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 -- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d6c0c5080 msgr2=0x7f9d6c1f49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d6c0c5080 0x7f9d6c1f49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6a59c640 1 -- 172.21.15.46:0/855405957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c1f4ef0 con 0x7f9d580039b0 2024-04-15T15:16:08.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d6ad9d640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d6c1509b0 0x7f9d6c0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.912+0000 7f9d69d9b640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d6c0c5080 0x7f9d6c1f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.913+0000 7f9d6a59c640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d6c0b4190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d64037040 tx=0x7f9d64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.913+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64042020 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.913+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6403a040 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.913+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c0b4790 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.913+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d64039070 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.913+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6c0b4d50 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.914+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c0690f0 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.914+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d64045020 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.915+0000 7f9d62ffd640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9d4c06c8f0 0x7f9d4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.915+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9d640ad540 con 0x7f9d580039b0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.915+0000 7f9d69d9b640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9d4c06c8f0 0x7f9d4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.916+0000 7f9d69d9b640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9d4c06c8f0 0x7f9d4c06edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d54007fd0 tx=0x7f9d54009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:08.919+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d6404c030 con 0x7f9d580039b0 2024-04-15T15:16:09.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.102+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9d6c06de90 con 0x7f9d580039b0 2024-04-15T15:16:09.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.102+0000 7f9d62ffd640 1 -- 172.21.15.46:0/855405957 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f9d6407b630 con 0x7f9d580039b0 2024-04-15T15:16:09.105 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9d4c06c8f0 msgr2=0x7f9d4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9d4c06c8f0 0x7f9d4c06edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d54007fd0 tx=0x7f9d54009040 comp rx=0 tx=0).stop 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 msgr2=0x7f9d6c0b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d6c0b4190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d64037040 tx=0x7f9d64004730 comp rx=0 tx=0).stop 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 shutdown_connections 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9d4c06c8f0 0x7f9d4c06edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9d6c1509b0 0x7f9d6c0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d6c0c5080 0x7f9d6c1f49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 --2- 172.21.15.46:0/855405957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d580039b0 0x7f9d6c0b4190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 >> 172.21.15.46:0/855405957 conn(0x7f9d6c09a9d0 msgr2=0x7f9d6c0cdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 shutdown_connections 2024-04-15T15:16:09.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.104+0000 7f9d70cb3640 1 -- 172.21.15.46:0/855405957 wait complete. 2024-04-15T15:16:09.118 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:09.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.225+0000 7f7875d1b640 1 Processor -- start 2024-04-15T15:16:09.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.225+0000 7f7875d1b640 1 -- start start 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7875d1b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787015c8d0 0x7f78700cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7875d1b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f787009b0e0 0x7f78700b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7875d1b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78700d0350 con 0x7f787009b650 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7875d1b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f78700b5b00 con 0x7f787009b0e0 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7875d1b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f78700ce100 con 0x7f787015c8d0 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f786ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f787009b0e0 0x7f78700b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7866ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f787009b650 0x7f787009a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55758/0 (socket says 172.21.15.46:55758) 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f7866ffd640 1 -- 172.21.15.46:0/2233833405 learned_addr learned my addr 172.21.15.46:0/2233833405 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.226+0000 7f786f7fe640 1 --2- 172.21.15.46:0/2233833405 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787015c8d0 0x7f78700cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.227+0000 7f786ffff640 1 -- 172.21.15.46:0/2233833405 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787015c8d0 msgr2=0x7f78700cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.227+0000 7f786ffff640 1 --2- 172.21.15.46:0/2233833405 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787015c8d0 0x7f78700cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.227+0000 7f786ffff640 1 -- 172.21.15.46:0/2233833405 >> v1:172.21.15.46:6789/0 conn(0x7f787009b650 legacy=0x7f787009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.227+0000 7f786ffff640 1 -- 172.21.15.46:0/2233833405 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78700b54b0 con 0x7f787009b0e0 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.227+0000 7f786ffff640 1 --2- 172.21.15.46:0/2233833405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700b5f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7868009d90 tx=0x7f786802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ac5f87f6b7e8ed server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f786effd640 1 -- 172.21.15.46:0/2233833405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868039070 con 0x7f787009b0e0 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f786effd640 1 -- 172.21.15.46:0/2233833405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786802cbc0 con 0x7f787009b0e0 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f786effd640 1 -- 172.21.15.46:0/2233833405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868034440 con 0x7f787009b0e0 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 -- 172.21.15.46:0/2233833405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 msgr2=0x7f78700b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 --2- 172.21.15.46:0/2233833405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700b5f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7868009d90 tx=0x7f786802be10 comp rx=0 tx=0).stop 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 -- 172.21.15.46:0/2233833405 shutdown_connections 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 --2- 172.21.15.46:0/2233833405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700b5f90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 --2- 172.21.15.46:0/2233833405 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787015c8d0 0x7f78700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 -- 172.21.15.46:0/2233833405 >> 172.21.15.46:0/2233833405 conn(0x7f78700b3fb0 msgr2=0x7f78700b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 -- 172.21.15.46:0/2233833405 shutdown_connections 2024-04-15T15:16:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.228+0000 7f7875d1b640 1 -- 172.21.15.46:0/2233833405 wait complete. 2024-04-15T15:16:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.230+0000 7f7875d1b640 1 Processor -- start 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.230+0000 7f7875d1b640 1 -- start start 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.230+0000 7f7875d1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.230+0000 7f7875d1b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787009b650 0x7f78701f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7875d1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c6270 0x7f787009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7875d1b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f78700b6650 con 0x7f787009b0e0 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7875d1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f78700b64d0 con 0x7f78700c6270 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7875d1b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f78700b67d0 con 0x7f787009b650 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42556/0 (socket says 172.21.15.46:42556) 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 -- 172.21.15.46:0/3061916321 learned_addr learned my addr 172.21.15.46:0/3061916321 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7866ffd640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787009b650 0x7f78701f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786ffff640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c6270 0x7f787009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 -- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787009b650 msgr2=0x7f78701f2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787009b650 0x7f78701f2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 -- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c6270 msgr2=0x7f787009edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c6270 0x7f787009edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 -- 172.21.15.46:0/3061916321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78701f34c0 con 0x7f787009b0e0 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786f7fe640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700c7870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f786000e8a0 tx=0x7f786000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860017070 con 0x7f787009b0e0 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786000c940 con 0x7f787009b0e0 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78700c67f0 con 0x7f787009b0e0 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786000cd30 con 0x7f787009b0e0 2024-04-15T15:16:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.231+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78700c6db0 con 0x7f787009b0e0 2024-04-15T15:16:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.233+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78700690f0 con 0x7f787009b0e0 2024-04-15T15:16:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.233+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f786001a070 con 0x7f787009b0e0 2024-04-15T15:16:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.233+0000 7f786cff9640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785006c8f0 0x7f785006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.233+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f786008bf90 con 0x7f787009b0e0 2024-04-15T15:16:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.234+0000 7f7866ffd640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785006c8f0 0x7f785006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.234+0000 7f7866ffd640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785006c8f0 0x7f785006edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7858007fd0 tx=0x7f7858007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.238+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78701f34c0 con 0x7f787009b0e0 2024-04-15T15:16:09.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.424+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78700b39e0 con 0x7f787009b0e0 2024-04-15T15:16:09.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.425+0000 7f786cff9640 1 -- 172.21.15.46:0/3061916321 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f7860059ac0 con 0x7f787009b0e0 2024-04-15T15:16:09.428 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785006c8f0 msgr2=0x7f785006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785006c8f0 0x7f785006edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7858007fd0 tx=0x7f7858007e80 comp rx=0 tx=0).stop 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 msgr2=0x7f78700c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700c7870 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f786000e8a0 tx=0x7f786000ed60 comp rx=0 tx=0).stop 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 shutdown_connections 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f785006c8f0 0x7f785006edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78700c6270 0x7f787009edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f787009b650 0x7f78701f2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 --2- 172.21.15.46:0/3061916321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f787009b0e0 0x7f78700c7870 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 >> 172.21.15.46:0/3061916321 conn(0x7f78700b3fb0 msgr2=0x7f7870070660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 shutdown_connections 2024-04-15T15:16:09.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.427+0000 7f7875d1b640 1 -- 172.21.15.46:0/3061916321 wait complete. 2024-04-15T15:16:09.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:09.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.551+0000 7fe14bc37640 1 Processor -- start 2024-04-15T15:16:09.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.552+0000 7fe14bc37640 1 -- start start 2024-04-15T15:16:09.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.552+0000 7fe14bc37640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14406b9e0 0x7fe1440705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.552+0000 7fe14bc37640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe14415ebc0 con 0x7fe1440aeed0 2024-04-15T15:16:09.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.552+0000 7fe14bc37640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1440ae150 con 0x7fe1440b0850 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.552+0000 7fe14bc37640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe144070c80 con 0x7fe14406b9e0 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe14a1d4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe1440aeed0 0x7fe1440af360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53396/0 (socket says 172.21.15.46:53396) 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe14a1d4640 1 -- 172.21.15.46:0/3896108927 learned_addr learned my addr 172.21.15.46:0/3896108927 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1499d3640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14406b9e0 0x7fe1440705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419571450 0 0) 0x7fe14415ebc0 con 0x7fe1440aeed0 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe130003520 con 0x7fe1440aeed0 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 119313152 0 0) 0x7fe1440ae150 con 0x7fe1440b0850 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe14415ebc0 con 0x7fe1440b0850 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1044053639 0 0) 0x7fe130003520 con 0x7fe1440aeed0 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14406b9e0 msgr2=0x7fe1440705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14406b9e0 0x7fe1440705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 >> v1:172.21.15.78:6789/0 conn(0x7fe1440b0850 legacy=0x7fe14406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1440ae3f0 con 0x7fe1440aeed0 2024-04-15T15:16:09.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fe13c0019f0 con 0x7fe1440aeed0 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 >> v1:172.21.15.46:6789/0 conn(0x7fe1440aeed0 legacy=0x7fe1440af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe130003b70 0x7fe130003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130004800 0x7fe130004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300051c0 0x7fe13000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe144070c80 con 0x7fe1300051c0 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe13000bfb0 con 0x7fe130004800 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe13000c1b0 con 0x7fe130003b70 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1499d3640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14406b9e0 0x7fe1440705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe14a1d4640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300051c0 0x7fe13000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1491d2640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130004800 0x7fe130004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.553+0000 7fe1499d3640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe130003b70 0x7fe130003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.554+0000 7fe14a1d4640 1 -- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe130003b70 msgr2=0x7fe130003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.554+0000 7fe14a1d4640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe130003b70 0x7fe130003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.554+0000 7fe14a1d4640 1 -- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130004800 msgr2=0x7fe130004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.554+0000 7fe14a1d4640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe130004800 0x7fe130004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.554+0000 7fe14a1d4640 1 -- 172.21.15.46:0/3896108927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe130003520 con 0x7fe1300051c0 2024-04-15T15:16:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.554+0000 7fe14a1d4640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300051c0 0x7fe13000b9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe13c00a7f0 tx=0x7fe130005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62ddf5c029262d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c00a410 con 0x7fe1300051c0 2024-04-15T15:16:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13c034630 con 0x7fe1300051c0 2024-04-15T15:16:09.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe1489d1640 1 -- 172.21.15.46:0/3896108927 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c02ce60 con 0x7fe1300051c0 2024-04-15T15:16:09.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14bc37640 1 -- 172.21.15.46:0/3896108927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300051c0 msgr2=0x7fe13000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14bc37640 1 --2- 172.21.15.46:0/3896108927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1300051c0 0x7fe13000b9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe13c00a7f0 tx=0x7fe130005830 comp rx=0 tx=0).stop 2024-04-15T15:16:09.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14a1d4640 1 -- 172.21.15.46:0/3896108927 reap_dead start 2024-04-15T15:16:09.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14bc37640 1 -- 172.21.15.46:0/3896108927 shutdown_connections 2024-04-15T15:16:09.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14bc37640 1 -- 172.21.15.46:0/3896108927 >> 172.21.15.46:0/3896108927 conn(0x7fe1440b2e70 msgr2=0x7fe1440b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14bc37640 1 -- 172.21.15.46:0/3896108927 shutdown_connections 2024-04-15T15:16:09.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.555+0000 7fe14bc37640 1 -- 172.21.15.46:0/3896108927 wait complete. 2024-04-15T15:16:09.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.557+0000 7fe14bc37640 1 Processor -- start 2024-04-15T15:16:09.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.557+0000 7fe14bc37640 1 -- start start 2024-04-15T15:16:09.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1300051c0 0x7fe144158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 0x7fe1440b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1440b4550 0x7fe1440b6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440cf0a0 con 0x7fe14409ec90 2024-04-15T15:16:09.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe1440ae9b0 con 0x7fe1300051c0 2024-04-15T15:16:09.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe1440705a0 con 0x7fe1440b4550 2024-04-15T15:16:09.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 0x7fe1440b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 0x7fe1440b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38526/0 (socket says 172.21.15.46:38526) 2024-04-15T15:16:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 -- 172.21.15.46:0/2202279540 learned_addr learned my addr 172.21.15.46:0/2202279540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14a1d4640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1440b4550 0x7fe1440b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1499d3640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1300051c0 0x7fe144158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 -- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1440b4550 msgr2=0x7fe1440b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1440b4550 0x7fe1440b6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 -- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1300051c0 msgr2=0x7fe144158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1300051c0 0x7fe144158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 -- 172.21.15.46:0/2202279540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1440b49d0 con 0x7fe14409ec90 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe1491d2640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 0x7fe1440b4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe13400f770 tx=0x7fe13400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13400fe80 con 0x7fe14409ec90 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe134018070 con 0x7fe14409ec90 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1441f4ca0 con 0x7fe14409ec90 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13400d730 con 0x7fe14409ec90 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.558+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1441f5260 con 0x7fe14409ec90 2024-04-15T15:16:09.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.559+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1440690f0 con 0x7fe14409ec90 2024-04-15T15:16:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.560+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe13400d8d0 con 0x7fe14409ec90 2024-04-15T15:16:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.561+0000 7fe13a7fc640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12806c920 0x7fe12806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.561+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe13408bce0 con 0x7fe14409ec90 2024-04-15T15:16:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.561+0000 7fe1499d3640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12806c920 0x7fe12806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.561+0000 7fe1499d3640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12806c920 0x7fe12806ede0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe144162090 tx=0x7fe140009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.565+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe134059780 con 0x7fe14409ec90 2024-04-15T15:16:09.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:09.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:09.757 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:09.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.752+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe14406fa00 con 0x7fe14409ec90 2024-04-15T15:16:09.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.753+0000 7fe13a7fc640 1 -- 172.21.15.46:0/2202279540 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fe13405da80 con 0x7fe14409ec90 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12806c920 msgr2=0x7fe12806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12806c920 0x7fe12806ede0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe144162090 tx=0x7fe140009190 comp rx=0 tx=0).stop 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 msgr2=0x7fe1440b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 0x7fe1440b4010 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe13400f770 tx=0x7fe13400fc30 comp rx=0 tx=0).stop 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 shutdown_connections 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12806c920 0x7fe12806ede0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe1440b4550 0x7fe1440b6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14409ec90 0x7fe1440b4010 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 --2- 172.21.15.46:0/2202279540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1300051c0 0x7fe144158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.755+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 >> 172.21.15.46:0/2202279540 conn(0x7fe1440b2e70 msgr2=0x7fe1440b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.756+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 shutdown_connections 2024-04-15T15:16:09.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.756+0000 7fe14bc37640 1 -- 172.21.15.46:0/2202279540 wait complete. 2024-04-15T15:16:09.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:09.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.913+0000 7fe2f92a5640 1 Processor -- start 2024-04-15T15:16:09.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f92a5640 1 -- start start 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f92a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2f415eb00 0x7fe2f40ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f92a5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b0560 0x7fe2f406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f92a5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f40cbd10 con 0x7fe2f40b09a0 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f92a5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f415eee0 con 0x7fe2f415eb00 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f92a5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f40cbff0 con 0x7fe2f40b0560 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.914+0000 7fe2f2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2f415eb00 0x7fe2f40ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2f40b09a0 0x7fe2f40ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53406/0 (socket says 172.21.15.46:53406) 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f37fe640 1 -- 172.21.15.46:0/502709016 learned_addr learned my addr 172.21.15.46:0/502709016 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2ebfff640 1 --2- 172.21.15.46:0/502709016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b0560 0x7fe2f406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/502709016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b0560 msgr2=0x7fe2f406b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f2ffd640 1 --2- 172.21.15.46:0/502709016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b0560 0x7fe2f406b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/502709016 >> v1:172.21.15.46:6789/0 conn(0x7fe2f40b09a0 legacy=0x7fe2f40ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/502709016 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f40cc2d0 con 0x7fe2f415eb00 2024-04-15T15:16:09.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.915+0000 7fe2f27fc640 1 -- 172.21.15.46:0/502709016 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268391511 0 0) 0x7fe2f40cbd10 con 0x7fe2f40b09a0 2024-04-15T15:16:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.916+0000 7fe2f2ffd640 1 --2- 172.21.15.46:0/502709016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f415eb00 0x7fe2f40ae2a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40097a0 tx=0x7fe2e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38140d47f8986ba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.916+0000 7fe2f27fc640 1 -- 172.21.15.46:0/502709016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4035020 con 0x7fe2f415eb00 2024-04-15T15:16:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.916+0000 7fe2f27fc640 1 -- 172.21.15.46:0/502709016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e4031830 con 0x7fe2f415eb00 2024-04-15T15:16:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.916+0000 7fe2f27fc640 1 -- 172.21.15.46:0/502709016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4031c00 con 0x7fe2f415eb00 2024-04-15T15:16:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 -- 172.21.15.46:0/502709016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f415eb00 msgr2=0x7fe2f40ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/502709016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f415eb00 0x7fe2f40ae2a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40097a0 tx=0x7fe2e402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 -- 172.21.15.46:0/502709016 shutdown_connections 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/502709016 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b0560 0x7fe2f406b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/502709016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f415eb00 0x7fe2f40ae2a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 -- 172.21.15.46:0/502709016 >> 172.21.15.46:0/502709016 conn(0x7fe2f409a9d0 msgr2=0x7fe2f40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 -- 172.21.15.46:0/502709016 shutdown_connections 2024-04-15T15:16:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.917+0000 7fe2f92a5640 1 -- 172.21.15.46:0/502709016 wait complete. 2024-04-15T15:16:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.918+0000 7fe2f92a5640 1 Processor -- start 2024-04-15T15:16:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.918+0000 7fe2f92a5640 1 -- start start 2024-04-15T15:16:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f92a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 0x7fe2f4152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f92a5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b09a0 0x7fe2f40b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f92a5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f415eb00 0x7fe2f40b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f92a5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f40ae7e0 con 0x7fe2f40b0560 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f92a5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f40cc890 con 0x7fe2f415eb00 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f92a5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f40cec60 con 0x7fe2f40b09a0 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 0x7fe2f4152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 0x7fe2f4152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38536/0 (socket says 172.21.15.46:38536) 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/3684632821 learned_addr learned my addr 172.21.15.46:0/3684632821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2ebfff640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b09a0 0x7fe2f40b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b09a0 msgr2=0x7fe2f40b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b09a0 0x7fe2f40b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f415eb00 msgr2=0x7fe2f40b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f415eb00 0x7fe2f40b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 -- 172.21.15.46:0/3684632821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f40b6ca0 con 0x7fe2f40b0560 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.919+0000 7fe2f2ffd640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 0x7fe2f4152cb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40098c0 tx=0x7fe2e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.920+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e403ba80 con 0x7fe2f40b0560 2024-04-15T15:16:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.920+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e403bc20 con 0x7fe2f40b0560 2024-04-15T15:16:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.920+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f40b6f80 con 0x7fe2f40b0560 2024-04-15T15:16:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.920+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40453f0 con 0x7fe2f40b0560 2024-04-15T15:16:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.920+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f40c6570 con 0x7fe2f40b0560 2024-04-15T15:16:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.921+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f40690f0 con 0x7fe2f40b0560 2024-04-15T15:16:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.921+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2e404c060 con 0x7fe2f40b0560 2024-04-15T15:16:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.922+0000 7fe2eb7fe640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2d406c8f0 0x7fe2d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.922+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe2e40adc40 con 0x7fe2f40b0560 2024-04-15T15:16:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.922+0000 7fe2ebfff640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2d406c8f0 0x7fe2d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.922+0000 7fe2ebfff640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2d406c8f0 0x7fe2d406edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40704a0 tx=0x7fe2dc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:09.926+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2e4040070 con 0x7fe2f40b0560 2024-04-15T15:16:10.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.151+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2f40ce1b0 con 0x7fe2f40b0560 2024-04-15T15:16:10.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.151+0000 7fe2eb7fe640 1 -- 172.21.15.46:0/3684632821 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7fe2e402f2e0 con 0x7fe2f40b0560 2024-04-15T15:16:10.154 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:10.154 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:10.154 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2d406c8f0 msgr2=0x7fe2d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2d406c8f0 0x7fe2d406edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe2f40704a0 tx=0x7fe2dc008040 comp rx=0 tx=0).stop 2024-04-15T15:16:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 msgr2=0x7fe2f4152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 0x7fe2f4152cb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40098c0 tx=0x7fe2e40046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 shutdown_connections 2024-04-15T15:16:10.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2d406c8f0 0x7fe2d406edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2f415eb00 0x7fe2f40b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2f40b09a0 0x7fe2f40b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 --2- 172.21.15.46:0/3684632821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2f40b0560 0x7fe2f4152cb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 >> 172.21.15.46:0/3684632821 conn(0x7fe2f409a9d0 msgr2=0x7fe2f40b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 shutdown_connections 2024-04-15T15:16:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.153+0000 7fe2f92a5640 1 -- 172.21.15.46:0/3684632821 wait complete. 2024-04-15T15:16:10.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.269+0000 7fd81d2a5640 1 Processor -- start 2024-04-15T15:16:10.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.269+0000 7fd81d2a5640 1 -- start start 2024-04-15T15:16:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd81d2a5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd81815c550 0x7fd8180affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd81d2a5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd81806e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd81d2a5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd81806c9f0 0x7fd81806ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd81d2a5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd8180cd800 con 0x7fd81815c550 2024-04-15T15:16:10.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd81d2a5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd8180ce380 con 0x7fd81806c9f0 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd81d2a5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd8180cda50 con 0x7fd8180b6400 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd816ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd81815c550 0x7fd8180affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.270+0000 7fd816ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd81815c550 0x7fd8180affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38542/0 (socket says 172.21.15.46:38542) 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd816ffd640 1 -- 172.21.15.46:0/4056219935 learned_addr learned my addr 172.21.15.46:0/4056219935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd8167fc640 1 --2- 172.21.15.46:0/4056219935 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd81806e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd8177fe640 1 --2- 172.21.15.46:0/4056219935 >> v2:172.21.15.78:3300/0 conn(0x7fd81806c9f0 0x7fd81806ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd816ffd640 1 -- 172.21.15.46:0/4056219935 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 msgr2=0x7fd81806e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd816ffd640 1 --2- 172.21.15.46:0/4056219935 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd81806e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd816ffd640 1 -- 172.21.15.46:0/4056219935 >> v2:172.21.15.78:3300/0 conn(0x7fd81806c9f0 msgr2=0x7fd81806ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd816ffd640 1 --2- 172.21.15.46:0/4056219935 >> v2:172.21.15.78:3300/0 conn(0x7fd81806c9f0 0x7fd81806ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd816ffd640 1 -- 172.21.15.46:0/4056219935 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8180cdd30 con 0x7fd81815c550 2024-04-15T15:16:10.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.271+0000 7fd8177fe640 1 --2- 172.21.15.46:0/4056219935 >> v2:172.21.15.78:3300/0 conn(0x7fd81806c9f0 0x7fd81806ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd816ffd640 1 --2- 172.21.15.46:0/4056219935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180affe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd818071f20 tx=0x7fd80802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa2b994b804b74c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd815ffb640 1 -- 172.21.15.46:0/4056219935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808035020 con 0x7fd81815c550 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd815ffb640 1 -- 172.21.15.46:0/4056219935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd808031870 con 0x7fd81815c550 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd815ffb640 1 -- 172.21.15.46:0/4056219935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808031c40 con 0x7fd81815c550 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 -- 172.21.15.46:0/4056219935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 msgr2=0x7fd8180affe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/4056219935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180affe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd818071f20 tx=0x7fd80802b910 comp rx=0 tx=0).stop 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 -- 172.21.15.46:0/4056219935 shutdown_connections 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/4056219935 >> v2:172.21.15.78:3300/0 conn(0x7fd81806c9f0 0x7fd81806ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/4056219935 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd81806e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/4056219935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180affe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 -- 172.21.15.46:0/4056219935 >> 172.21.15.46:0/4056219935 conn(0x7fd818072440 msgr2=0x7fd818072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 -- 172.21.15.46:0/4056219935 shutdown_connections 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.272+0000 7fd81d2a5640 1 -- 172.21.15.46:0/4056219935 wait complete. 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.274+0000 7fd81d2a5640 1 Processor -- start 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.274+0000 7fd81d2a5640 1 -- start start 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.274+0000 7fd81d2a5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd81806c9f0 0x7fd8180c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.274+0000 7fd81d2a5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd8180ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.274+0000 7fd81d2a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.274+0000 7fd81d2a5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd81815c930 con 0x7fd81815c550 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd81d2a5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd8180b6120 con 0x7fd81806c9f0 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd81d2a5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd81815cab0 con 0x7fd8180b6400 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38546/0 (socket says 172.21.15.46:38546) 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 -- 172.21.15.46:0/3463409602 learned_addr learned my addr 172.21.15.46:0/3463409602 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8167fc640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd8180ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd816ffd640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd81806c9f0 0x7fd8180c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 -- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 msgr2=0x7fd8180ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd8180ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 -- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd81806c9f0 msgr2=0x7fd8180c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd81806c9f0 0x7fd8180c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 -- 172.21.15.46:0/3463409602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8180b4a40 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd8177fe640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180cd130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd81000cef0 tx=0x7fd81000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81001a070 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd81000d840 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8180b4d20 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81000dc30 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.275+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8180b52e0 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.276+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8180690f0 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.277+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd81001d070 con 0x7fd81815c550 2024-04-15T15:16:10.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.277+0000 7fd7ff7fe640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd80406c8f0 0x7fd80406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.277+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd81008c5a0 con 0x7fd81815c550 2024-04-15T15:16:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.278+0000 7fd816ffd640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd80406c8f0 0x7fd80406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.278+0000 7fd816ffd640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd80406c8f0 0x7fd80406edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd808036000 tx=0x7fd80803c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.282+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd81005a040 con 0x7fd81815c550 2024-04-15T15:16:10.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.483+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8180732e0 con 0x7fd81815c550 2024-04-15T15:16:10.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.484+0000 7fd7ff7fe640 1 -- 172.21.15.46:0/3463409602 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18466 (secure 0 0 0) 0x7fd81005e340 con 0x7fd81815c550 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:15:42.956268+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd80406c8f0 msgr2=0x7fd80406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd80406c8f0 0x7fd80406edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd808036000 tx=0x7fd80803c040 comp rx=0 tx=0).stop 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 msgr2=0x7fd8180cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180cd130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd81000cef0 tx=0x7fd81000a710 comp rx=0 tx=0).stop 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 shutdown_connections 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd80406c8f0 0x7fd80406edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81815c550 0x7fd8180cd130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8180b6400 0x7fd8180ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 --2- 172.21.15.46:0/3463409602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd81806c9f0 0x7fd8180c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 >> 172.21.15.46:0/3463409602 conn(0x7fd818072440 msgr2=0x7fd81815cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 shutdown_connections 2024-04-15T15:16:10.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.486+0000 7fd81d2a5640 1 -- 172.21.15.46:0/3463409602 wait complete. 2024-04-15T15:16:10.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:10.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.603+0000 7f1ba7b32640 1 Processor -- start 2024-04-15T15:16:10.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.603+0000 7f1ba7b32640 1 -- start start 2024-04-15T15:16:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba7b32640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ba0072d80 0x7f1ba0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba7b32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba00af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba7b32640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ba015db40 con 0x7f1ba00af3e0 2024-04-15T15:16:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba7b32640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba015cfc0 con 0x7f1ba0072d80 2024-04-15T15:16:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba7b32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba015d2a0 con 0x7f1ba00d0da0 2024-04-15T15:16:10.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ba0072d80 0x7f1ba0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ba0072d80 0x7f1ba0071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38548/0 (socket says 172.21.15.46:38548) 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2510870916 learned_addr learned my addr 172.21.15.46:0/2510870916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba60cf640 1 --2- 172.21.15.46:0/2510870916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba00af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2510870916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 msgr2=0x7f1ba00af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 --2- 172.21.15.46:0/2510870916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba00af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2510870916 >> v1:172.21.15.78:6789/0 conn(0x7f1ba00af3e0 legacy=0x7f1ba00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.604+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2510870916 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba015d580 con 0x7f1ba0072d80 2024-04-15T15:16:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.605+0000 7f1ba58ce640 1 --2- 172.21.15.46:0/2510870916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba0071d10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009870 tx=0x7f1b9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54750142c9790399 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.605+0000 7f1ba48cc640 1 -- 172.21.15.46:0/2510870916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c035020 con 0x7f1ba0072d80 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.605+0000 7f1ba48cc640 1 -- 172.21.15.46:0/2510870916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c031640 con 0x7f1ba0072d80 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.605+0000 7f1ba48cc640 1 -- 172.21.15.46:0/2510870916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c02d040 con 0x7f1ba0072d80 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2510870916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 msgr2=0x7f1ba0071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2510870916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba0071d10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009870 tx=0x7f1b9c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2510870916 shutdown_connections 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2510870916 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba00af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2510870916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba0071d10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2510870916 >> 172.21.15.46:0/2510870916 conn(0x7f1ba006be00 msgr2=0x7f1ba006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2510870916 shutdown_connections 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.606+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2510870916 wait complete. 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.607+0000 7f1ba7b32640 1 Processor -- start 2024-04-15T15:16:10.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.607+0000 7f1ba7b32640 1 -- start start 2024-04-15T15:16:10.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba00cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba00af3e0 0x7f1ba0157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba0159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba015cd10 con 0x7f1ba0072d80 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba015cb90 con 0x7f1ba00af3e0 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba006e060 con 0x7f1ba00d0da0 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba00cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba00cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38550/0 (socket says 172.21.15.46:38550) 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2020108399 learned_addr learned my addr 172.21.15.46:0/2020108399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba60cf640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba0159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba50cd640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba00af3e0 0x7f1ba0157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 msgr2=0x7f1ba0159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba0159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba00af3e0 msgr2=0x7f1ba0157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba00af3e0 0x7f1ba0157c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 -- 172.21.15.46:0/2020108399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba0158560 con 0x7f1ba0072d80 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba50cd640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba00af3e0 0x7f1ba0157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba58ce640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba00cf0b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c02b930 tx=0x7f1b9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c040070 con 0x7f1ba0072d80 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c03e070 con 0x7f1ba0072d80 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba0158840 con 0x7f1ba0072d80 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.608+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c035040 con 0x7f1ba0072d80 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.609+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba0158e00 con 0x7f1ba0072d80 2024-04-15T15:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.609+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ba00689e0 con 0x7f1ba0072d80 2024-04-15T15:16:10.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.610+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b9c038020 con 0x7f1ba0072d80 2024-04-15T15:16:10.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.610+0000 7f1b967fc640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b8406c8f0 0x7f1b8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.611+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1b9c0ad760 con 0x7f1ba0072d80 2024-04-15T15:16:10.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.611+0000 7f1ba50cd640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b8406c8f0 0x7f1b8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.611+0000 7f1ba50cd640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b8406c8f0 0x7f1b8406edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00af1e0 tx=0x7f1b90009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.615+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b9c07b850 con 0x7f1ba0072d80 2024-04-15T15:16:10.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.837+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ba0073300 con 0x7f1ba0072d80 2024-04-15T15:16:10.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.838+0000 7f1b967fc640 1 -- 172.21.15.46:0/2020108399 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f1b9c07b150 con 0x7f1ba0072d80 2024-04-15T15:16:10.841 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:10.841 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:10.841 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:10.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b8406c8f0 msgr2=0x7f1b8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b8406c8f0 0x7f1b8406edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ba00af1e0 tx=0x7f1b90009040 comp rx=0 tx=0).stop 2024-04-15T15:16:10.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 msgr2=0x7f1ba00cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba00cf0b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c02b930 tx=0x7f1b9c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 shutdown_connections 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1b8406c8f0 0x7f1b8406edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1ba00d0da0 0x7f1ba0159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ba00af3e0 0x7f1ba0157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 --2- 172.21.15.46:0/2020108399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ba0072d80 0x7f1ba00cf0b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 >> 172.21.15.46:0/2020108399 conn(0x7f1ba006be00 msgr2=0x7f1ba0071f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 shutdown_connections 2024-04-15T15:16:10.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:10.840+0000 7f1ba7b32640 1 -- 172.21.15.46:0/2020108399 wait complete. 2024-04-15T15:16:11.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:15:37.926112+0000', 'modified': '2024-04-15T15:16:00.995766+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4284}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4284': {'gid': 4284, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.78:6837/3854702652', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6835', 'nonce': 3854702652}, {'type': 'v1', 'addr': '172.21.15.78:6837', 'nonce': 3854702652}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4284, 'qdb_cluster': [4284]} 2024-04-15T15:16:11.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:16:11.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-15T15:16:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.364+0000 7f56ecd00640 1 Processor -- start 2024-04-15T15:16:11.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56ecd00640 1 -- start start 2024-04-15T15:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56ecd00640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f56e815ed00 0x7f56e8070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56ecd00640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56e80cf600 0x7f56e80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56ecd00640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e80ae440 con 0x7f56e806ef70 2024-04-15T15:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56ecd00640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f56e80ae720 con 0x7f56e80cf600 2024-04-15T15:16:11.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56ecd00640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f56e80aea00 con 0x7f56e815ed00 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.365+0000 7f56dffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56e80cf600 0x7f56e80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56dffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56e80cf600 0x7f56e80cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38560/0 (socket says 172.21.15.46:38560) 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56dffff640 1 -- 172.21.15.46:0/791864571 learned_addr learned my addr 172.21.15.46:0/791864571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56e659c640 1 --2- 172.21.15.46:0/791864571 >> v2:172.21.15.78:3300/0 conn(0x7f56e815ed00 0x7f56e8070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56dffff640 1 -- 172.21.15.46:0/791864571 >> v2:172.21.15.78:3300/0 conn(0x7f56e815ed00 msgr2=0x7f56e8070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56dffff640 1 --2- 172.21.15.46:0/791864571 >> v2:172.21.15.78:3300/0 conn(0x7f56e815ed00 0x7f56e8070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56dffff640 1 -- 172.21.15.46:0/791864571 >> v1:172.21.15.78:6789/0 conn(0x7f56e806ef70 legacy=0x7f56e806f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.366+0000 7f56dffff640 1 -- 172.21.15.46:0/791864571 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e80aece0 con 0x7f56e80cf600 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56dffff640 1 --2- 172.21.15.46:0/791864571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e80cf600 0x7f56e80cfa90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56e80b4110 tx=0x7f56d002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535f9d2d0712d220 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56e5d9b640 1 -- 172.21.15.46:0/791864571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0034070 con 0x7f56e80cf600 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56e5d9b640 1 -- 172.21.15.46:0/791864571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d0030860 con 0x7f56e80cf600 2024-04-15T15:16:11.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56e5d9b640 1 -- 172.21.15.46:0/791864571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0030c30 con 0x7f56e80cf600 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 -- 172.21.15.46:0/791864571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e80cf600 msgr2=0x7f56e80cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 --2- 172.21.15.46:0/791864571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e80cf600 0x7f56e80cfa90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f56e80b4110 tx=0x7f56d002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 -- 172.21.15.46:0/791864571 shutdown_connections 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 --2- 172.21.15.46:0/791864571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e80cf600 0x7f56e80cfa90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 --2- 172.21.15.46:0/791864571 >> v2:172.21.15.78:3300/0 conn(0x7f56e815ed00 0x7f56e8070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 -- 172.21.15.46:0/791864571 >> 172.21.15.46:0/791864571 conn(0x7f56e809a9d0 msgr2=0x7f56e80d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 -- 172.21.15.46:0/791864571 shutdown_connections 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.367+0000 7f56ecd00640 1 -- 172.21.15.46:0/791864571 wait complete. 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 Processor -- start 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 -- start start 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f56e806ef70 0x7f56e80b5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56e80cf600 0x7f56e80b6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 0x7f56e815d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56e8070e60 con 0x7f56e815ed00 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f56e80701a0 con 0x7f56e80cf600 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.369+0000 7f56ecd00640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f56e8070fe0 con 0x7f56e806ef70 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 0x7f56e815d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 0x7f56e815d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38564/0 (socket says 172.21.15.46:38564) 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 -- 172.21.15.46:0/4233134662 learned_addr learned my addr 172.21.15.46:0/4233134662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e659c640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f56e806ef70 0x7f56e80b5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56dffff640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56e80cf600 0x7f56e80b6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 -- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f56e806ef70 msgr2=0x7f56e80b5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f56e806ef70 0x7f56e80b5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 -- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56e80cf600 msgr2=0x7f56e80b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56e80cf600 0x7f56e80b6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 -- 172.21.15.46:0/4233134662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e80b42c0 con 0x7f56e815ed00 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 0x7f56e815d0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f56e000c6e0 tx=0x7f56e000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56deffd640 1 -- 172.21.15.46:0/4233134662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0003b70 con 0x7f56e815ed00 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56deffd640 1 -- 172.21.15.46:0/4233134662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e0003d30 con 0x7f56e815ed00 2024-04-15T15:16:11.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56deffd640 1 -- 172.21.15.46:0/4233134662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0005310 con 0x7f56e815ed00 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e80b45a0 con 0x7f56e815ed00 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.370+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e80b4b60 con 0x7f56e815ed00 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.372+0000 7f56deffd640 1 -- 172.21.15.46:0/4233134662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f56e00054b0 con 0x7f56e815ed00 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.372+0000 7f56deffd640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f56c806c8f0 0x7f56c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.372+0000 7f56deffd640 1 -- 172.21.15.46:0/4233134662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56e008bb10 con 0x7f56e815ed00 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.372+0000 7f56e659c640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f56c806c8f0 0x7f56c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.373+0000 7f56e659c640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f56c806c8f0 0x7f56c806edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009b40 tx=0x7f56d8009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.373+0000 7f56ef9bf740 1 Processor -- start 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.373+0000 7f56ef9bf740 1 -- start start 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 0x55f8cced3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf011f0 0x55f8ccf03ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8cced42f0 0x55f8ccf040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f8cce3e9c0 con 0x55f8ccf00e10 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f8ccde5650 con 0x55f8cced42f0 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55f8cce3a980 con 0x55f8ccf011f0 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 0x55f8cced3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 0x55f8cced3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38580/0 (socket says 172.21.15.46:38580) 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 -- 172.21.15.46:0/2743915537 learned_addr learned my addr 172.21.15.46:0/2743915537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56dffff640 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf011f0 0x55f8ccf03ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8cced42f0 0x55f8ccf040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 -- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf011f0 msgr2=0x55f8ccf03ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf011f0 0x55f8ccf03ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 -- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8cced42f0 msgr2=0x55f8ccf040e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8cced42f0 0x55f8ccf040e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.374+0000 7f56e659c640 1 -- 172.21.15.46:0/2743915537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ccb512d0 con 0x55f8ccf00e10 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56e659c640 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 0x55f8cced3db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f56d8011ee0 tx=0x7f56d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38967d67ddad3568 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56c7fff640 1 -- 172.21.15.46:0/2743915537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d801b070 con 0x55f8ccf00e10 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56c7fff640 1 -- 172.21.15.46:0/2743915537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d8027070 con 0x55f8ccf00e10 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56c7fff640 1 -- 172.21.15.46:0/2743915537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d8003a40 con 0x55f8ccf00e10 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 -- 172.21.15.46:0/2743915537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 msgr2=0x55f8cced3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 0x55f8cced3db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f56d8011ee0 tx=0x7f56d8014000 comp rx=0 tx=0).stop 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 -- 172.21.15.46:0/2743915537 shutdown_connections 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8cced42f0 0x55f8ccf040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf011f0 0x55f8ccf03ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/2743915537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf00e10 0x55f8cced3db0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.375+0000 7f56ef9bf740 1 -- 172.21.15.46:0/2743915537 >> 172.21.15.46:0/2743915537 conn(0x55f8cced6020 msgr2=0x55f8ccf1eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.376+0000 7f56ef9bf740 1 -- 172.21.15.46:0/2743915537 shutdown_connections 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.376+0000 7f56ef9bf740 1 -- 172.21.15.46:0/2743915537 wait complete. 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.376+0000 7f56ef9bf740 1 -- start start 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.376+0000 7f56ef9bf740 1 Processor -- start 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf31d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf07210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf07750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f8cced85c0 con 0x55f8ccf011f0 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f8ccdf8a10 con 0x55f8ccefeab0 2024-04-15T15:16:11.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55f8cced79b0 con 0x55f8ccf00e10 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf07210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56ef9bf740 10 client.0 _ll_register_callbacks cb 0x55f8cce34250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56dffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf07210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38592/0 (socket says 172.21.15.46:38592) 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56dffff640 1 -- 172.21.15.46:0/3559371186 learned_addr learned my addr 172.21.15.46:0/3559371186 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56e659c640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf31d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.377+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf07750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56dffff640 1 -- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 msgr2=0x55f8ccf31d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56dffff640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf31d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56dffff640 1 -- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 msgr2=0x55f8ccf07750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56dffff640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf07750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56dffff640 1 -- 172.21.15.46:0/3559371186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ccc1b6b0 con 0x55f8ccf011f0 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf07750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:11.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56dffff640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf07210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f56d00090b0 tx=0x7f56d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a86d8dbec26c96 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56c57fa640 1 -- 172.21.15.46:0/3559371186 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0034040 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56c57fa640 1 -- 172.21.15.46:0/3559371186 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56d0043890 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56d8027070 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56c57fa640 1 -- 172.21.15.46:0/3559371186 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f56d0052050 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 10 client.4797 fetch_fsmap learned FSMap version 8 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55f8ccf04f10 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56c57fa640 1 -- 172.21.15.46:0/3559371186 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d8027070 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56c57fa640 1 -- 172.21.15.46:0/3559371186 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f56d0055020 con 0x55f8ccf011f0 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 10 client.4797 fetch_fsmap finished waiting for FSMap version 8 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 10 client.4797 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 20 client.4797 populate_metadata read hostname 'smithi046' 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf16430 0x55f8ccf05d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 4 client.4797 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.378+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 --> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8cce4c3e0 con 0x55f8ccf16430 2024-04-15T15:16:11.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.379+0000 7f56e659c640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf16430 0x55f8ccf05d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.379+0000 7f56e659c640 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf16430 0x55f8ccf05d50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5537573af3a3c610 server_cookie=718a7dc0b06397da in_seq=0 out_seq=0 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.379+0000 7f56c57fa640 10 client.4797 ms_handle_connect on v2:172.21.15.78:6835/3854702652 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56c57fa640 1 -- 172.21.15.46:0/3559371186 <== mds.0 v2:172.21.15.78:6835/3854702652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8cce4c3e0 con 0x55f8ccf16430 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56c57fa640 10 client.4797 handle_command_reply: tid=0 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 client.4797 shutdown 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 msgr2=0x55f8ccf07210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf07210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f56d00090b0 tx=0x7f56d0004730 comp rx=0 tx=0).stop 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 shutdown_connections 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf16430 0x55f8ccf05d50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf07750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf07210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/3559371186 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf31d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56c57fa640 0 client.4797 ms_handle_reset on v2:172.21.15.78:6835/3854702652 2024-04-15T15:16:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 >> 172.21.15.46:0/3559371186 conn(0x55f8cced42f0 msgr2=0x55f8ccf035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 shutdown_connections 2024-04-15T15:16:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 1 -- 172.21.15.46:0/3559371186 wait complete. 2024-04-15T15:16:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.380+0000 7f56ef9bf740 20 client.4797 trim_cache size 0 max 16384 2024-04-15T15:16:11.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.398+0000 7f56ef9bf740 1 Processor -- start 2024-04-15T15:16:11.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.398+0000 7f56ef9bf740 1 -- start start 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 0x55f8cced3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf30fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccf011f0 0x55f8cced85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56e82011b0 con 0x55f8ccefeab0 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f56e8200510 con 0x55f8ccf011f0 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f56e8200b60 con 0x55f8ccf00e10 2024-04-15T15:16:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 0x55f8cced3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 0x55f8cced3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38608/0 (socket says 172.21.15.46:38608) 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 -- 172.21.15.46:0/1020414892 learned_addr learned my addr 172.21.15.46:0/1020414892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 -- 172.21.15.46:0/1020414892 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 msgr2=0x55f8ccf30fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf30fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 -- 172.21.15.46:0/1020414892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccf011f0 msgr2=0x55f8cced85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccf011f0 0x55f8cced85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 -- 172.21.15.46:0/1020414892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ccecfda0 con 0x55f8ccefeab0 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.399+0000 7f56e659c640 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 0x55f8cced3ad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f56d8011eb0 tx=0x7f56d8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9cc4f5881e65be server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56c77fe640 1 -- 172.21.15.46:0/1020414892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d803e070 con 0x55f8ccefeab0 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56c77fe640 1 -- 172.21.15.46:0/1020414892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d8027040 con 0x55f8ccefeab0 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56c77fe640 1 -- 172.21.15.46:0/1020414892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d801b040 con 0x55f8ccefeab0 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 -- 172.21.15.46:0/1020414892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 msgr2=0x55f8cced3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 0x55f8cced3ad0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f56d8011eb0 tx=0x7f56d8016000 comp rx=0 tx=0).stop 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 -- 172.21.15.46:0/1020414892 shutdown_connections 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccf011f0 0x55f8cced85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf30fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/1020414892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccefeab0 0x55f8cced3ad0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 -- 172.21.15.46:0/1020414892 >> 172.21.15.46:0/1020414892 conn(0x55f8cced42f0 msgr2=0x55f8ccedadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 -- 172.21.15.46:0/1020414892 shutdown_connections 2024-04-15T15:16:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.400+0000 7f56ef9bf740 1 -- 172.21.15.46:0/1020414892 wait complete. 2024-04-15T15:16:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.401+0000 7f56ef9bf740 1 -- start start 2024-04-15T15:16:11.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.401+0000 7f56ef9bf740 1 Processor -- start 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf14530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf17b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf18070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f8ccf162b0 con 0x55f8ccf011f0 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55f8ccf064b0 con 0x55f8ccefeab0 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55f8ccf168d0 con 0x55f8ccf00e10 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 10 client.0 _ll_register_callbacks cb 0x7f56d0030a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf18070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf18070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38614/0 (socket says 172.21.15.46:38614) 2024-04-15T15:16:11.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 -- 172.21.15.46:0/494290359 learned_addr learned my addr 172.21.15.46:0/494290359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e659c640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf14530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56dffff640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf17b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 -- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 msgr2=0x55f8ccf17b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf17b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 -- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 msgr2=0x55f8ccf14530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf14530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 -- 172.21.15.46:0/494290359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ccefa590 con 0x55f8ccf011f0 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56dffff640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e659c640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf14530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56e6d9d640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf18070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56e000a430 tx=0x7f56e0059750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88513a16f836e705 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56c57fa640 1 -- 172.21.15.46:0/494290359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e0055cb0 con 0x55f8ccf011f0 2024-04-15T15:16:11.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56c57fa640 1 -- 172.21.15.46:0/494290359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f56e0058a90 con 0x55f8ccf011f0 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56d8027040 con 0x55f8ccf011f0 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56c57fa640 1 -- 172.21.15.46:0/494290359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f56e0076050 con 0x55f8ccf011f0 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 10 client.4803 fetch_fsmap learned FSMap version 8 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55f8ccf170d0 con 0x55f8ccf011f0 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.402+0000 7f56c57fa640 1 -- 172.21.15.46:0/494290359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56d8027040 con 0x55f8ccf011f0 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56c57fa640 1 -- 172.21.15.46:0/494290359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7f56e0055390 con 0x55f8ccf011f0 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 10 client.4803 fetch_fsmap finished waiting for FSMap version 8 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 10 client.4803 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 20 client.4803 populate_metadata read hostname 'smithi046' 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf98e50 0x55f8ccf99230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 4 client.4803 mds_command: new command op to 4284 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 --> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] -- command(tid 0: {"prefix": "status"}) -- 0x55f8cce46250 con 0x55f8ccf98e50 2024-04-15T15:16:11.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56e659c640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf98e50 0x55f8ccf99230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56c57fa640 10 client.4803 ms_handle_connect on v2:172.21.15.78:6835/3854702652 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56e659c640 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf98e50 0x55f8ccf99230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d4a1d192451380a server_cookie=a71369eb3ad2977f in_seq=0 out_seq=0 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56c57fa640 1 -- 172.21.15.46:0/494290359 <== mds.0 v2:172.21.15.78:6835/3854702652 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55f8cce46250 con 0x55f8ccf98e50 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56c57fa640 10 client.4803 handle_command_reply: tid=0 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 1 client.4803 shutdown 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 msgr2=0x55f8ccf18070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.403+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf18070 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f56e000a430 tx=0x7f56e0059750 comp rx=0 tx=0).stop 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 shutdown_connections 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:6835/3854702652,v1:172.21.15.78:6837/3854702652] conn(0x55f8ccf98e50 0x55f8ccf99230 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f8ccf011f0 0x55f8ccf18070 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55f8ccf00e10 0x55f8ccf17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56c57fa640 0 client.4803 ms_handle_reset on v2:172.21.15.78:6835/3854702652 2024-04-15T15:16:11.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 --2- 172.21.15.46:0/494290359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f8ccefeab0 0x55f8ccf14530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 >> 172.21.15.46:0/494290359 conn(0x55f8cced42f0 msgr2=0x55f8cced3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 shutdown_connections 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 1 -- 172.21.15.46:0/494290359 wait complete. 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ef9bf740 20 client.4803 trim_cache size 0 max 16384 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "95f1e89e-6559-430a-8622-bd018dcc0afd", 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4284, 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-15T15:16:11.415 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-15T15:16:11.416 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 32.850797786999998, 2024-04-15T15:16:11.416 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 8, 2024-04-15T15:16:11.416 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 25, 2024-04-15T15:16:11.416 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 25, 2024-04-15T15:16:11.416 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 39.428822676999999 2024-04-15T15:16:11.416 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f56c806c8f0 msgr2=0x7f56c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f56c806c8f0 0x7f56c806edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009b40 tx=0x7f56d8009190 comp rx=0 tx=0).stop 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 msgr2=0x7f56e815d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 0x7f56e815d0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f56e000c6e0 tx=0x7f56e000cba0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 shutdown_connections 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f56c806c8f0 0x7f56c806edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56e815ed00 0x7f56e815d0c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56e80cf600 0x7f56e80b6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 --2- 172.21.15.46:0/4233134662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f56e806ef70 0x7f56e80b5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 >> 172.21.15.46:0/4233134662 conn(0x7f56e809a9d0 msgr2=0x7f56e80b39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 shutdown_connections 2024-04-15T15:16:11.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.404+0000 7f56ecd00640 1 -- 172.21.15.46:0/4233134662 wait complete. 2024-04-15T15:16:11.419 INFO:teuthology.run_tasks:Running task sequential... 2024-04-15T15:16:11.429 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-15T15:16:11.430 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-15T15:16:11.430 INFO:teuthology.task.print:Enabling mgr modules 2024-04-15T15:16:11.430 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-15T15:16:11.449 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-15T15:16:11.450 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-15T15:16:11.450 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-15T15:16:11.450 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:16:11.450 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:11.450 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-15T15:16:11.450 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:16:11.450 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:11.450 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-15T15:16:11.450 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:16:11.450 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:11.451 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-15T15:16:11.451 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:16:11.451 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:11.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:11.451 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:16:11.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:11.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-04-15T15:16:11.548 INFO:teuthology.orchestra.run.smithi078.stderr:Cannot find device "ceph-brx" 2024-04-15T15:16:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:16:11.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:11.550 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:16:11.566 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-15T15:16:11.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:11.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:16:11.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:11.636 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-04-15T15:16:11.731 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-15T15:16:11.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.834+0000 7f8afabbd640 1 Processor -- start 2024-04-15T15:16:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.834+0000 7f8afabbd640 1 -- start start 2024-04-15T15:16:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8afabbd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8af415e910 0x7f8af40d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8afabbd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8afabbd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8af415ecf0 con 0x7f8af40b6110 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8afabbd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f8af40d21d0 con 0x7f8af415e910 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8afabbd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8af40d24b0 con 0x7f8af40cfda0 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8af3fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8af8959640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8af415e910 0x7f8af40d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8af915a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8af40b6110 0x7f8af406cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:44654/0 (socket says 172.21.15.46:44654) 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8af915a640 1 -- 172.21.15.46:0/1435832570 learned_addr learned my addr 172.21.15.46:0/1435832570 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.835+0000 7f8af8959640 1 -- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 msgr2=0x7f8af40d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.836+0000 7f8af8959640 1 --2- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.836+0000 7f8af8959640 1 -- 172.21.15.46:0/1435832570 >> v1:172.21.15.78:6789/0 conn(0x7f8af40b6110 legacy=0x7f8af406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.836+0000 7f8af8959640 1 -- 172.21.15.46:0/1435832570 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af40d2790 con 0x7f8af415e910 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.836+0000 7f8af37fe640 1 -- 172.21.15.46:0/1435832570 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3022677031 0 0) 0x7f8af415ecf0 con 0x7f8af40b6110 2024-04-15T15:16:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.836+0000 7f8af3fff640 1 --2- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.836+0000 7f8af8959640 1 --2- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af415e910 0x7f8af40d3270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ae40097a0 tx=0x7f8ae402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b0f12cbc8fb9971 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8af37fe640 1 -- 172.21.15.46:0/1435832570 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4035020 con 0x7f8af415e910 2024-04-15T15:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8af37fe640 1 -- 172.21.15.46:0/1435832570 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ae4031830 con 0x7f8af415e910 2024-04-15T15:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8af37fe640 1 -- 172.21.15.46:0/1435832570 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4031c00 con 0x7f8af415e910 2024-04-15T15:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 -- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af415e910 msgr2=0x7f8af40d3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 --2- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af415e910 0x7f8af40d3270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ae40097a0 tx=0x7f8ae402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 -- 172.21.15.46:0/1435832570 shutdown_connections 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 --2- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 --2- 172.21.15.46:0/1435832570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af415e910 0x7f8af40d3270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 -- 172.21.15.46:0/1435832570 >> 172.21.15.46:0/1435832570 conn(0x7f8af408bf60 msgr2=0x7f8af40d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 -- 172.21.15.46:0/1435832570 shutdown_connections 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.837+0000 7f8afabbd640 1 -- 172.21.15.46:0/1435832570 wait complete. 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.838+0000 7f8afabbd640 1 Processor -- start 2024-04-15T15:16:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.838+0000 7f8afabbd640 1 -- start start 2024-04-15T15:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8afabbd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40b6110 0x7f8af41f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8afabbd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8afabbd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 0x7f8af406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8afabbd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8af40d37b0 con 0x7f8af415e910 2024-04-15T15:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8afabbd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8af40d2d50 con 0x7f8af40b6110 2024-04-15T15:16:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8afabbd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8af40d5470 con 0x7f8af40cfda0 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8af915a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 0x7f8af406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8af915a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 0x7f8af406dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38624/0 (socket says 172.21.15.46:38624) 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.839+0000 7f8af915a640 1 -- 172.21.15.46:0/4102574131 learned_addr learned my addr 172.21.15.46:0/4102574131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af8959640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40b6110 0x7f8af41f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af3fff640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af915a640 1 -- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 msgr2=0x7f8af40cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af915a640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af915a640 1 -- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40b6110 msgr2=0x7f8af41f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af915a640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40b6110 0x7f8af41f6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af915a640 1 -- 172.21.15.46:0/4102574131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af40ba2e0 con 0x7f8af415e910 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af8959640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40b6110 0x7f8af41f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af915a640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 0x7f8af406dee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c7d0 tx=0x7f8aec00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec018070 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aec00a640 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af40b9d20 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec00aa70 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.840+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af406d6e0 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.841+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af40689e0 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.842+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8aec01b070 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.842+0000 7f8af17fa640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8ae006c8f0 0x7f8ae006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.842+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8aec08b490 con 0x7f8af415e910 2024-04-15T15:16:11.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.842+0000 7f8af8959640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8ae006c8f0 0x7f8ae006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.843+0000 7f8af8959640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8ae006c8f0 0x7f8ae006edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4036000 tx=0x7f8ae4008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:11.846+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8aec059580 con 0x7f8af415e910 2024-04-15T15:16:12.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.070+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8af40b6e60 con 0x7f8af415e910 2024-04-15T15:16:12.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.070+0000 7f8af17fa640 1 -- 172.21.15.46:0/4102574131 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f8aec058e80 con 0x7f8af415e910 2024-04-15T15:16:12.073 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8ae006c8f0 msgr2=0x7f8ae006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8ae006c8f0 0x7f8ae006edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4036000 tx=0x7f8ae4008030 comp rx=0 tx=0).stop 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 msgr2=0x7f8af406dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 0x7f8af406dee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00c7d0 tx=0x7f8aec00cc90 comp rx=0 tx=0).stop 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 shutdown_connections 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8ae006c8f0 0x7f8ae006edb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8af415e910 0x7f8af406dee0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8af40cfda0 0x7f8af40cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 --2- 172.21.15.46:0/4102574131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40b6110 0x7f8af41f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 >> 172.21.15.46:0/4102574131 conn(0x7f8af408bf60 msgr2=0x7f8af40d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 shutdown_connections 2024-04-15T15:16:12.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12.072+0000 7f8afabbd640 1 -- 172.21.15.46:0/4102574131 wait complete. 2024-04-15T15:16:12.085 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:12.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:12.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-15T15:16:12.085 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:16:12.085 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:12.086 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:12.086 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:12.086 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:12.086 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:12.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:12.086 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42680sec preferred_lft 42680sec 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-15T15:16:12.104 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:16:12.104 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:12.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:12.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:12.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:12.255 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:12.282 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-15T15:16:12.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:12.283 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-15T15:16:12.341 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-15T15:16:12.341 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:12.341 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:16:12.341 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:16:12.341 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:12.341 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:12.341 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:12.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:16:12.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:16:12.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:16:12.342 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:12.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:12.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:12.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:12.480 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:16:12.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:12.496 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-04-15T15:16:12.551 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:12.551 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:12.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:12.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-15T15:16:12.551 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:12.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:12.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:12.675 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-15T15:16:12.675 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:12.675 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:12.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-15T15:16:12.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:16:12.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-15T15:16:12.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-15T15:16:12.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-15T15:16:12.676 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:12.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:12.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:12.805 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:12.805 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:12.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-04-15T15:16:12.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:16:12.805 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:12.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:12.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:12.889 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:16:12.889 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:12.890 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:12.905 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:12.905 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:12.905 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:12.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:16:13.029 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:16:13.085 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:16:13.086 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:13.087 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:13.088 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:16:13.088 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:13.088 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:13.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:13.088 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:16:13.141 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:13.141 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:16:13.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:16:13.213 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:16:13.270 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:13.271 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:13.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:13.272 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:16:13.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-04-15T15:16:13.271+0000 7f768423e480 -1 init, newargv = 0x55b9fa128420 newargc=15 2024-04-15T15:16:13.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[47608]: starting ceph client 2024-04-15T15:16:13.278 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[47608]: starting fuse 2024-04-15T15:16:13.288 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:16:13.289 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:13.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:13.289 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-04-15T15:16:13.290 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-04-15T15:16:13.291 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-04-15T15:16:13.411 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.47608.asok 2024-04-15T15:16:13.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:13.422 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47608.asok 2024-04-15T15:16:13.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:13.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47608.asok status 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "47608", 2024-04-15T15:16:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4812, 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4812 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1228087398 2024-04-15T15:16:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1228087398 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4812 192.168.0.1:0/1228087398", 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1228087398", 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 8, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 25, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-04-15T15:16:13.563 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:16:13.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.672+0000 7f754f3e8640 1 Processor -- start 2024-04-15T15:16:13.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.673+0000 7f754f3e8640 1 -- start start 2024-04-15T15:16:13.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.673+0000 7f754f3e8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.673+0000 7f754f3e8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75480b35f0 0x7f75480b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.673+0000 7f754f3e8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75480d5190 con 0x7f75480b44f0 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.673+0000 7f754f3e8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f75480cf3c0 con 0x7f75480b35f0 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.673+0000 7f754f3e8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f75480cf6a0 con 0x7f7548161fe0 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75480b35f0 0x7f75480b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75480b35f0 0x7f75480b3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38632/0 (socket says 172.21.15.46:38632) 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 -- 172.21.15.46:0/2286885826 learned_addr learned my addr 172.21.15.46:0/2286885826 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754d184640 1 --2- 172.21.15.46:0/2286885826 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 -- 172.21.15.46:0/2286885826 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 msgr2=0x7f75480d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 --2- 172.21.15.46:0/2286885826 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 -- 172.21.15.46:0/2286885826 >> v1:172.21.15.78:6789/0 conn(0x7f75480b44f0 legacy=0x7f75480d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.674+0000 7f754c983640 1 -- 172.21.15.46:0/2286885826 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75480cf980 con 0x7f75480b35f0 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754c983640 1 --2- 172.21.15.46:0/2286885826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f75480b3fb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f753c009870 tx=0x7f753c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc2f1d0301c354 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f7547fff640 1 -- 172.21.15.46:0/2286885826 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c034070 con 0x7f75480b35f0 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f7547fff640 1 -- 172.21.15.46:0/2286885826 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f753c030840 con 0x7f75480b35f0 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f7547fff640 1 -- 172.21.15.46:0/2286885826 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c030c10 con 0x7f75480b35f0 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 -- 172.21.15.46:0/2286885826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 msgr2=0x7f75480b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 --2- 172.21.15.46:0/2286885826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f75480b3fb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f753c009870 tx=0x7f753c02b980 comp rx=0 tx=0).stop 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 -- 172.21.15.46:0/2286885826 shutdown_connections 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 --2- 172.21.15.46:0/2286885826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f75480b3fb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 --2- 172.21.15.46:0/2286885826 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 -- 172.21.15.46:0/2286885826 >> 172.21.15.46:0/2286885826 conn(0x7f754808bf60 msgr2=0x7f75480d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 -- 172.21.15.46:0/2286885826 shutdown_connections 2024-04-15T15:16:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.675+0000 7f754f3e8640 1 -- 172.21.15.46:0/2286885826 wait complete. 2024-04-15T15:16:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 Processor -- start 2024-04-15T15:16:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 -- start start 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f7548151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75480b44f0 0x7f7548160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75480d4d60 con 0x7f75480b35f0 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75481623c0 con 0x7f75480b44f0 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754f3e8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f75480d4ee0 con 0x7f7548161fe0 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754d184640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f7548151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754d184640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f7548151ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38636/0 (socket says 172.21.15.46:38636) 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.677+0000 7f754d184640 1 -- 172.21.15.46:0/3960717409 learned_addr learned my addr 172.21.15.46:0/3960717409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d985640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754c983640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75480b44f0 0x7f7548160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d184640 1 -- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 msgr2=0x7f75480ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d184640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d184640 1 -- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75480b44f0 msgr2=0x7f7548160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d184640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75480b44f0 0x7f7548160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d184640 1 -- 172.21.15.46:0/3960717409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7548152000 con 0x7f75480b35f0 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d985640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754c983640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75480b44f0 0x7f7548160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.678+0000 7f754d184640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f7548151ac0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f753800efc0 tx=0x7f753800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.680+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538018020 con 0x7f75480b35f0 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.680+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7538014480 con 0x7f75480b35f0 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.680+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75480cd200 con 0x7f75480b35f0 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.680+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538014850 con 0x7f75480b35f0 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.680+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75480cd7c0 con 0x7f75480b35f0 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.681+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75480689e0 con 0x7f75480b35f0 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.682+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7538005d10 con 0x7f75480b35f0 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.682+0000 7f7545ffb640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f753406c8f0 0x7f753406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.682+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f753808c930 con 0x7f75480b35f0 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.682+0000 7f754c983640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f753406c8f0 0x7f753406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.683+0000 7f754c983640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f753406c8f0 0x7f753406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f753c036000 tx=0x7f753c03b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.686+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7538022070 con 0x7f75480b35f0 2024-04-15T15:16:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.913+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f754808c6d0 con 0x7f75480b35f0 2024-04-15T15:16:13.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.913+0000 7f7545ffb640 1 -- 172.21.15.46:0/3960717409 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f7538010040 con 0x7f75480b35f0 2024-04-15T15:16:13.915 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:16:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f753406c8f0 msgr2=0x7f753406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f753406c8f0 0x7f753406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f753c036000 tx=0x7f753c03b040 comp rx=0 tx=0).stop 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 msgr2=0x7f7548151ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f7548151ac0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f753800efc0 tx=0x7f753800c490 comp rx=0 tx=0).stop 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 shutdown_connections 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f753406c8f0 0x7f753406edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7548161fe0 0x7f75480ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75480b44f0 0x7f7548160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 --2- 172.21.15.46:0/3960717409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75480b35f0 0x7f7548151ac0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.915+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 >> 172.21.15.46:0/3960717409 conn(0x7f754808bf60 msgr2=0x7f75480b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.916+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 shutdown_connections 2024-04-15T15:16:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:13.916+0000 7f754f3e8640 1 -- 172.21.15.46:0/3960717409 wait complete. 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:13.928 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:13.929 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:13.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:13.929 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-15T15:16:13.945 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42678sec preferred_lft 42678sec 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 1e:a1:9f:32:6c:b0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::50d2:bff:fe30:917d/64 scope link tentative 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 1e:a1:9f:32:6c:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::1ca1:9fff:fe32:6cb0/64 scope link tentative 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:13.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:13.948 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:14.012 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-15T15:16:14.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:14.013 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-15T15:16:14.070 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-15T15:16:14.070 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:14.070 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:16:14.070 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:16:14.070 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:14.070 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:14.070 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:14.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:16:14.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:16:14.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:16:14.071 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:14.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:14.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:14.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:14.210 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:16:14.225 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:14.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:14.226 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-04-15T15:16:14.281 INFO:teuthology.orchestra.run.smithi046.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-15T15:16:14.281 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:14.281 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:14.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:14.281 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-15T15:16:14.281 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:14.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:14.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:14.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:14.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:16:14.434 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:14.434 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:14.435 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:14.435 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.435 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:14.435 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.435 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:14.435 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether f2:31:c6:9b:f2:66 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:16:14.436 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:16:14.436 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.436 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::f031:c6ff:fe9b:f266/64 scope link tentative 2024-04-15T15:16:14.436 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:14.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:16:14.502 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether f2:31:c6:9b:f2:66 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::f031:c6ff:fe9b:f266/64 scope link tentative 2024-04-15T15:16:14.503 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:14.504 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-15T15:16:14.505 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:14.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:14.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:14.677 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:14.677 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:16:14.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.1 up 2024-04-15T15:16:14.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:16:14.677 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:16:14.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:14.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:14.762 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:16:14.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:14.762 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:14.785 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:14.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:14.785 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:14.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:16:14.909 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:14.966 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:16:14.966 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:16:14.966 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:16:14.966 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:16:14.967 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:14.968 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:14.969 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:14.969 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:14.969 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:14.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:14.969 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:16:15.022 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:16:15.022 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:16:15.022 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-15T15:16:15.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:16:15.094 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:15.152 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:16:15.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:16:15.152 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:16:15.152 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:16:15.152 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:16:15.153 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:16:15.153 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:16:15.153 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:16:15.153 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:16:15.154 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:15.155 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:15.156 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:15.156 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:15.156 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:15.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:15.156 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:16:15.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:2024-04-15T15:16:15.152+0000 7f62b7c3c480 -1 init, newargv = 0x56416de59d20 newargc=15 2024-04-15T15:16:15.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[48106]: starting ceph client 2024-04-15T15:16:15.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[48106]: starting fuse 2024-04-15T15:16:15.172 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:16:15.172 INFO:teuthology.orchestra.run.smithi046.stdout:44 2024-04-15T15:16:15.172 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-04-15T15:16:15.173 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:15.174 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:16:15.175 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.2")) 2024-04-15T15:16:15.175 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-04-15T15:16:15.295 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.2.48106.asok 2024-04-15T15:16:15.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:15.305 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.48106.asok 2024-04-15T15:16:15.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:15.305 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.48106.asok status 2024-04-15T15:16:15.446 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:16:15.446 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-04-15T15:16:15.446 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "2", 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "48106", 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4824, 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-04-15T15:16:15.447 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4824 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1085398873 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:16:15.448 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1085398873 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4824 192.168.0.2:0/1085398873", 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.2:0/1085398873", 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 8, 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 25, 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-04-15T15:16:15.449 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:16:15.455 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.557+0000 7fe151159640 1 Processor -- start 2024-04-15T15:16:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.557+0000 7fe151159640 1 -- start start 2024-04-15T15:16:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe151159640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe151159640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe14c15ecf0 con 0x7fe14c0b6820 2024-04-15T15:16:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe151159640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe14c0d21d0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe151159640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe14c0d24b0 con 0x7fe14c15e910 2024-04-15T15:16:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14b59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe14c0b6820 0x7fe14c06cc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53414/0 (socket says 172.21.15.46:53414) 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14b59e640 1 -- 172.21.15.46:0/2669084185 learned_addr learned my addr 172.21.15.46:0/2669084185 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14ad9d640 1 --2- 172.21.15.46:0/2669084185 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14a59c640 1 -- 172.21.15.46:0/2669084185 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543724184 0 0) 0x7fe14c15ecf0 con 0x7fe14c0b6820 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14a59c640 1 -- 172.21.15.46:0/2669084185 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe138003520 con 0x7fe14c0b6820 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14a59c640 1 -- 172.21.15.46:0/2669084185 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 198699304 0 0) 0x7fe14c0d21d0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14ad9d640 1 -- 172.21.15.46:0/2669084185 >> v1:172.21.15.78:6789/0 conn(0x7fe14c0cfda0 legacy=0x7fe14c0b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14ad9d640 1 -- 172.21.15.46:0/2669084185 >> v1:172.21.15.46:6789/0 conn(0x7fe14c0b6820 legacy=0x7fe14c06cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.558+0000 7fe14ad9d640 1 -- 172.21.15.46:0/2669084185 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14c0d2790 con 0x7fe14c15e910 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.559+0000 7fe14ad9d640 1 --2- 172.21.15.46:0/2669084185 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0d3270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe13c0097d0 tx=0x7fe13c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46fa45b2963cfc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.559+0000 7fe14a59c640 1 -- 172.21.15.46:0/2669084185 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c035020 con 0x7fe14c15e910 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.559+0000 7fe14a59c640 1 -- 172.21.15.46:0/2669084185 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13c031890 con 0x7fe14c15e910 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.559+0000 7fe14a59c640 1 -- 172.21.15.46:0/2669084185 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13c031c40 con 0x7fe14c15e910 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 -- 172.21.15.46:0/2669084185 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 msgr2=0x7fe14c0d3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 --2- 172.21.15.46:0/2669084185 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0d3270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe13c0097d0 tx=0x7fe13c02b930 comp rx=0 tx=0).stop 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 -- 172.21.15.46:0/2669084185 shutdown_connections 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 --2- 172.21.15.46:0/2669084185 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0d3270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 -- 172.21.15.46:0/2669084185 >> 172.21.15.46:0/2669084185 conn(0x7fe14c08bf60 msgr2=0x7fe14c0d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 -- 172.21.15.46:0/2669084185 shutdown_connections 2024-04-15T15:16:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.560+0000 7fe151159640 1 -- 172.21.15.46:0/2669084185 wait complete. 2024-04-15T15:16:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.561+0000 7fe151159640 1 Processor -- start 2024-04-15T15:16:15.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.561+0000 7fe151159640 1 -- start start 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe151159640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c0b6820 0x7fe14c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe151159640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 0x7fe14c151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe151159640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0b8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe151159640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c0d37b0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe151159640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe14c0d2d50 con 0x7fe14c0b6820 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe151159640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe14c0d5470 con 0x7fe14c15e910 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 0x7fe14c151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 0x7fe14c151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38652/0 (socket says 172.21.15.46:38652) 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 -- 172.21.15.46:0/2755984778 learned_addr learned my addr 172.21.15.46:0/2755984778 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe14b59e640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0b8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe14ad9d640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c0b6820 0x7fe14c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 -- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 msgr2=0x7fe14c0b8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0b8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 -- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c0b6820 msgr2=0x7fe14c157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c0b6820 0x7fe14c157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe143fff640 1 -- 172.21.15.46:0/2755984778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14c0ccd70 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.562+0000 7fe14b59e640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.563+0000 7fe143fff640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 0x7fe14c151860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c4e0 tx=0x7fe13400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.563+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe134017070 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.563+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1340035f0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.563+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe14c0b95f0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.563+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340039e0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.563+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe14c0b9bb0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.564+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe14c0689e0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.564+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe13401a070 con 0x7fe14c0cfda0 2024-04-15T15:16:15.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.565+0000 7fe1437fe640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12c06c8f0 0x7fe12c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.565+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe13408b120 con 0x7fe14c0cfda0 2024-04-15T15:16:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.565+0000 7fe14ad9d640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12c06c8f0 0x7fe12c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.566+0000 7fe14ad9d640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12c06c8f0 0x7fe12c06edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe13c036000 tx=0x7fe13c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.569+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe134058bc0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.795+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe14c0b6e60 con 0x7fe14c0cfda0 2024-04-15T15:16:15.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.795+0000 7fe1437fe640 1 -- 172.21.15.46:0/2755984778 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fe13405cec0 con 0x7fe14c0cfda0 2024-04-15T15:16:15.798 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:16:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12c06c8f0 msgr2=0x7fe12c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12c06c8f0 0x7fe12c06edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe13c036000 tx=0x7fe13c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 msgr2=0x7fe14c151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 0x7fe14c151860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe13400c4e0 tx=0x7fe13400c9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 shutdown_connections 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe12c06c8f0 0x7fe12c06edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe14c15e910 0x7fe14c0b8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe14c0cfda0 0x7fe14c151860 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 --2- 172.21.15.46:0/2755984778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe14c0b6820 0x7fe14c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.797+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 >> 172.21.15.46:0/2755984778 conn(0x7fe14c08bf60 msgr2=0x7fe14c0d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.798+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 shutdown_connections 2024-04-15T15:16:15.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:15.798+0000 7fe151159640 1 -- 172.21.15.46:0/2755984778 wait complete. 2024-04-15T15:16:15.810 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:15.810 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:15.810 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-04-15T15:16:15.810 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:16:15.810 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:15.811 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:15.811 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:15.811 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:15.811 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:15.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:15.811 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-04-15T15:16:15.831 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:15.831 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:15.831 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:15.831 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:15.831 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:15.831 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42662sec preferred_lft 42662sec 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-04-15T15:16:15.832 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:15.833 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:16:15.833 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:15.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:16.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:16.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:16.039 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:16.069 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-04-15T15:16:16.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:16.071 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-04-15T15:16:16.139 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-04-15T15:16:16.139 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:16.140 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:16.140 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:16.140 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:16.140 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:16.140 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:16.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:16.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:16.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:16.140 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:16.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:16.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:16.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:16.448 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:16:16.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:16.463 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-04-15T15:16:16.519 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:16.519 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:16.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:16.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-15T15:16:16.519 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:16.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:16.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:16.647 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-15T15:16:16.648 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:16.648 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:16.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-15T15:16:16.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:16:16.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-15T15:16:16.648 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-15T15:16:16.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-15T15:16:16.649 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:16.798 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:16.798 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:16.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-04-15T15:16:16.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:16:16.798 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:16.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:16.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:16.882 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:16:16.882 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:16.882 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:16.897 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:16.898 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:16.898 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:16.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:16:17.022 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:17.078 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:16:17.079 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:17.080 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:17.081 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:17.081 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:16:17.081 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:17.081 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:17.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.081 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:16:17.133 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:17.134 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:16:17.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:16:17.203 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:17.259 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:17.260 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:17.261 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:17.262 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:16:17.262 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:17.262 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:17.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.262 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:16:17.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:2024-04-15T15:16:17.264+0000 7fbdc2840480 -1 init, newargv = 0x556c45966a70 newargc=15 2024-04-15T15:16:17.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[43556]: starting ceph client 2024-04-15T15:16:17.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[43556]: starting fuse 2024-04-15T15:16:17.282 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:16:17.282 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-15T15:16:17.282 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:17.282 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-04-15T15:16:17.283 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.1")) 2024-04-15T15:16:17.284 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-04-15T15:16:17.404 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.1.43556.asok 2024-04-15T15:16:17.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:17.415 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43556.asok 2024-04-15T15:16:17.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43556.asok status 2024-04-15T15:16:17.555 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-04-15T15:16:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-04-15T15:16:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "1", 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "43556", 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4598, 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4598 2024-04-15T15:16:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1314544351 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1314544351 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.4598 192.168.0.1:0/1314544351", 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/1314544351", 2024-04-15T15:16:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-04-15T15:16:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 8, 2024-04-15T15:16:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 25, 2024-04-15T15:16:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-04-15T15:16:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-04-15T15:16:17.558 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-04-15T15:16:17.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.680+0000 7eff063e9640 1 Processor -- start 2024-04-15T15:16:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.680+0000 7eff063e9640 1 -- start start 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.680+0000 7eff063e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7eff00161fe0 0x7eff000d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff063e9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff000d1060 0x7eff000d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff063e9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff000d5190 con 0x7eff000b35f0 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff063e9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7eff000d19d0 con 0x7eff000d1060 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff063e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7eff000cf3c0 con 0x7eff00161fe0 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff04986640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff000d1060 0x7eff000d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7efef7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7eff000b35f0 0x7eff000b3fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53416/0 (socket says 172.21.15.46:53416) 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7efef7fff640 1 -- 172.21.15.46:0/2125588844 learned_addr learned my addr 172.21.15.46:0/2125588844 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7efefffff640 1 --2- 172.21.15.46:0/2125588844 >> v2:172.21.15.78:3300/0 conn(0x7eff00161fe0 0x7eff000d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff04986640 1 -- 172.21.15.46:0/2125588844 >> v2:172.21.15.78:3300/0 conn(0x7eff00161fe0 msgr2=0x7eff000d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff04986640 1 --2- 172.21.15.46:0/2125588844 >> v2:172.21.15.78:3300/0 conn(0x7eff00161fe0 0x7eff000d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff04986640 1 -- 172.21.15.46:0/2125588844 >> v1:172.21.15.46:6789/0 conn(0x7eff000b35f0 legacy=0x7eff000b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7eff04986640 1 -- 172.21.15.46:0/2125588844 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff000cf6a0 con 0x7eff000d1060 2024-04-15T15:16:17.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.681+0000 7efeff7fe640 1 -- 172.21.15.46:0/2125588844 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331383852 0 0) 0x7eff000d5190 con 0x7eff000b35f0 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.682+0000 7eff04986640 1 --2- 172.21.15.46:0/2125588844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff000d1490 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff0008cdc0 tx=0x7efef802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93207e8b282817a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.682+0000 7efeff7fe640 1 -- 172.21.15.46:0/2125588844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8039070 con 0x7eff000d1060 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.682+0000 7efeff7fe640 1 -- 172.21.15.46:0/2125588844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef802cc80 con 0x7eff000d1060 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.682+0000 7efeff7fe640 1 -- 172.21.15.46:0/2125588844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8034660 con 0x7eff000d1060 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 -- 172.21.15.46:0/2125588844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 msgr2=0x7eff000d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 --2- 172.21.15.46:0/2125588844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff000d1490 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff0008cdc0 tx=0x7efef802bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 -- 172.21.15.46:0/2125588844 shutdown_connections 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 --2- 172.21.15.46:0/2125588844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff000d1490 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 --2- 172.21.15.46:0/2125588844 >> v2:172.21.15.78:3300/0 conn(0x7eff00161fe0 0x7eff000d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 -- 172.21.15.46:0/2125588844 >> 172.21.15.46:0/2125588844 conn(0x7eff0008bf60 msgr2=0x7eff000d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:17.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 -- 172.21.15.46:0/2125588844 shutdown_connections 2024-04-15T15:16:17.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.683+0000 7eff063e9640 1 -- 172.21.15.46:0/2125588844 wait complete. 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.684+0000 7eff063e9640 1 Processor -- start 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.684+0000 7eff063e9640 1 -- start start 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff000b35f0 0x7eff0015c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff001f5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7eff00161fe0 0x7eff000ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7eff000d5470 con 0x7eff000d1060 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7eff000b46c0 con 0x7eff000b35f0 2024-04-15T15:16:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7eff001623c0 con 0x7eff00161fe0 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff001f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff001f5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38660/0 (socket says 172.21.15.46:38660) 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 -- 172.21.15.46:0/1208250888 learned_addr learned my addr 172.21.15.46:0/1208250888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efefffff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff000b35f0 0x7eff0015c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff04986640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7eff00161fe0 0x7eff000ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 -- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7eff00161fe0 msgr2=0x7eff000ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7eff00161fe0 0x7eff000ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 -- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff000b35f0 msgr2=0x7eff0015c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff000b35f0 0x7eff0015c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 -- 172.21.15.46:0/1208250888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff000cc9c0 con 0x7eff000d1060 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efefffff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff000b35f0 0x7eff0015c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efef7fff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff001f5670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efee800c500 tx=0x7efee800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8018070 con 0x7eff000d1060 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee8007580 con 0x7eff000d1060 2024-04-15T15:16:17.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.685+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff000ccca0 con 0x7eff000d1060 2024-04-15T15:16:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.686+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8004a20 con 0x7eff000d1060 2024-04-15T15:16:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.686+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff001f5bb0 con 0x7eff000d1060 2024-04-15T15:16:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.687+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff000689e0 con 0x7eff000d1060 2024-04-15T15:16:17.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.687+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efee801b070 con 0x7eff000d1060 2024-04-15T15:16:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.688+0000 7efefd7fa640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7efee006c8f0 0x7efee006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.688+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efee808b0b0 con 0x7eff000d1060 2024-04-15T15:16:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.688+0000 7efefffff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7efee006c8f0 0x7efee006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.688+0000 7efefffff640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7efee006c8f0 0x7efee006edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efef0009e90 tx=0x7efef0009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.692+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efee8058be0 con 0x7eff000d1060 2024-04-15T15:16:17.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.916+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7eff0008e100 con 0x7eff000d1060 2024-04-15T15:16:17.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.916+0000 7efefd7fa640 1 -- 172.21.15.46:0/1208250888 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7efee805cee0 con 0x7eff000d1060 2024-04-15T15:16:17.918 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7efee006c8f0 msgr2=0x7efee006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7efee006c8f0 0x7efee006edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efef0009e90 tx=0x7efef0009190 comp rx=0 tx=0).stop 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 msgr2=0x7eff001f5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff001f5670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efee800c500 tx=0x7efee800c9c0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 shutdown_connections 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7efee006c8f0 0x7efee006edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7eff00161fe0 0x7eff000ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff000d1060 0x7eff001f5670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 --2- 172.21.15.46:0/1208250888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff000b35f0 0x7eff0015c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 >> 172.21.15.46:0/1208250888 conn(0x7eff0008bf60 msgr2=0x7eff0006d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:17.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 shutdown_connections 2024-04-15T15:16:17.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:17.918+0000 7eff063e9640 1 -- 172.21.15.46:0/1208250888 wait complete. 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:17.931 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:17.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.931 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.948 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42660sec preferred_lft 42660sec 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.949 INFO:teuthology.orchestra.run.smithi078.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 9e:e5:17:fe:02:aa brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::60de:b7ff:fe5c:109c/64 scope link 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 9e:e5:17:fe:02:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::9ce5:17ff:fefe:2aa/64 scope link tentative 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.951 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:18.016 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-04-15T15:16:18.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.017 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-04-15T15:16:18.074 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-04-15T15:16:18.074 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:18.074 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:18.074 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:18.074 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:18.074 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:18.075 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:18.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:18.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:18.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:18.075 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:18.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:18.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:18.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.213 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:16:18.228 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:18.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.229 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2024-04-15T15:16:18.285 INFO:teuthology.orchestra.run.smithi078.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-15T15:16:18.286 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:18.286 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:18.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:18.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-15T15:16:18.286 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:18.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:18.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:18.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:16:18.439 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether d2:e5:49:cb:1f:5f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::d0e5:49ff:fecb:1f5f/64 scope link tentative 2024-04-15T15:16:18.440 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.442 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether d2:e5:49:cb:1f:5f brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:16:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::d0e5:49ff:fecb:1f5f/64 scope link 2024-04-15T15:16:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.512 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-15T15:16:18.512 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:18.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:18.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:18.681 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:18.681 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:16:18.682 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.1 up 2024-04-15T15:16:18.682 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:16:18.682 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:16:18.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:18.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:18.770 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:16:18.770 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:18.770 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:18.785 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:18.785 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:18.786 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:18.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:16:18.910 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:18.967 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:16:18.967 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:16:18.967 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:16:18.967 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:16:18.967 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:16:18.967 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:18.968 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:18.969 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:18.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.970 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:16:19.023 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:16:19.023 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-04-15T15:16:19.024 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-15T15:16:19.066 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:16:19.094 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:19.152 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:16:19.152 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:16:19.152 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:16:19.152 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:16:19.152 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:16:19.153 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:16:19.153 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:16:19.153 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:16:19.153 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:16:19.153 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:16:19.153 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:16:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:16:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:16:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:16:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:16:19.154 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:16:19.155 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:19.155 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:19.155 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:19.155 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:16:19.155 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:19.155 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:19.156 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:16:19.156 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:16:19.156 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:19.156 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:19.156 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:19.156 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:19.157 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:16:19.157 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:19.157 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:19.157 INFO:teuthology.orchestra.run.smithi078.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.157 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:19.157 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:19.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.158 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:16:19.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:2024-04-15T15:16:19.152+0000 7f778483e480 -1 init, newargv = 0x561d47e42ee0 newargc=15 2024-04-15T15:16:19.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:ceph-fuse[44013]: starting ceph client 2024-04-15T15:16:19.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:ceph-fuse[44013]: starting fuse 2024-04-15T15:16:19.176 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:16:19.176 INFO:teuthology.orchestra.run.smithi078.stdout:44 2024-04-15T15:16:19.177 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-04-15T15:16:19.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:19.177 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-04-15T15:16:19.177 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-04-15T15:16:19.177 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:16:19.178 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-04-15T15:16:19.179 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:19.180 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:19.180 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:19.180 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-04-15T15:16:19.180 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-04-15T15:16:19.180 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-04-15T15:16:19.180 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:19.181 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:16:19.181 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.3")) 2024-04-15T15:16:19.181 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-04-15T15:16:19.298 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.3.44013.asok 2024-04-15T15:16:19.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:16:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:19.309 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44013.asok 2024-04-15T15:16:19.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44013.asok status 2024-04-15T15:16:19.447 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-04-15T15:16:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-04-15T15:16:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "3", 2024-04-15T15:16:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-04-15T15:16:19.448 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:16:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "44013", 2024-04-15T15:16:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-04-15T15:16:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-04-15T15:16:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:19.449 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 4607, 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4607 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:19.450 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2196422234 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2196422234 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.4607 192.168.0.2:0/2196422234", 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.2:0/2196422234", 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 8, 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 26, 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:19.451 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-04-15T15:16:19.452 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-04-15T15:16:19.452 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-04-15T15:16:19.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.456 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-04-15T15:16:19.456 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:19.476 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-04-15T15:16:19.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:19.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.476 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:19.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.550 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-04-15T15:16:19.550 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:19.609 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-04-15T15:16:19.610 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:19.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:19.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.681 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-04-15T15:16:19.681 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.698 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-04-15T15:16:19.698 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.768 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-04-15T15:16:19.768 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:19.826 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-04-15T15:16:19.826 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:19.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.826 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:19.895 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-15T15:16:19.910 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.018+0000 7f0182967640 1 Processor -- start 2024-04-15T15:16:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.018+0000 7f0182967640 1 -- start start 2024-04-15T15:16:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0182967640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0182967640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f017c0b7be0 0x7f017c0d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0182967640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f017c0d1b20 0x7f017c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0182967640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f017c0d3790 con 0x7f017c0b7be0 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0182967640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f017c0d45b0 con 0x7f017c0d1b20 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0182967640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f017c15eb70 con 0x7f017c15e790 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f017c0b7be0 0x7f017c0d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f017c0b7be0 0x7f017c0d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36804/0 (socket says 172.21.15.46:36804) 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 -- 172.21.15.46:0/3659299863 learned_addr learned my addr 172.21.15.46:0/3659299863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017bfff640 1 --2- 172.21.15.46:0/3659299863 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f0180f04640 1 --2- 172.21.15.46:0/3659299863 >> v2:172.21.15.78:3300/0 conn(0x7f017c0d1b20 0x7f017c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 -- 172.21.15.46:0/3659299863 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 msgr2=0x7f017c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 --2- 172.21.15.46:0/3659299863 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 -- 172.21.15.46:0/3659299863 >> v2:172.21.15.78:3300/0 conn(0x7f017c0d1b20 msgr2=0x7f017c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 --2- 172.21.15.46:0/3659299863 >> v2:172.21.15.78:3300/0 conn(0x7f017c0d1b20 0x7f017c0d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.019+0000 7f017b7fe640 1 -- 172.21.15.46:0/3659299863 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c0d3c80 con 0x7f017c0b7be0 2024-04-15T15:16:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.020+0000 7f017b7fe640 1 --2- 172.21.15.46:0/3659299863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c0d15e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f017c0cebe0 tx=0x7f017002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1eeea52b0fab70d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.020+0000 7f017affd640 1 -- 172.21.15.46:0/3659299863 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170038070 con 0x7f017c0b7be0 2024-04-15T15:16:20.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.020+0000 7f017affd640 1 -- 172.21.15.46:0/3659299863 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f017002f050 con 0x7f017c0b7be0 2024-04-15T15:16:20.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.020+0000 7f017affd640 1 -- 172.21.15.46:0/3659299863 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0170031970 con 0x7f017c0b7be0 2024-04-15T15:16:20.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 -- 172.21.15.46:0/3659299863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 msgr2=0x7f017c0d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 --2- 172.21.15.46:0/3659299863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c0d15e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f017c0cebe0 tx=0x7f017002b7f0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 -- 172.21.15.46:0/3659299863 shutdown_connections 2024-04-15T15:16:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 --2- 172.21.15.46:0/3659299863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c0d15e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 --2- 172.21.15.46:0/3659299863 >> v2:172.21.15.78:3300/0 conn(0x7f017c0d1b20 0x7f017c0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 --2- 172.21.15.46:0/3659299863 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 -- 172.21.15.46:0/3659299863 >> 172.21.15.46:0/3659299863 conn(0x7f017c067b40 msgr2=0x7f017c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 -- 172.21.15.46:0/3659299863 shutdown_connections 2024-04-15T15:16:20.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.021+0000 7f0182967640 1 -- 172.21.15.46:0/3659299863 wait complete. 2024-04-15T15:16:20.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.022+0000 7f0182967640 1 Processor -- start 2024-04-15T15:16:20.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.022+0000 7f0182967640 1 -- start start 2024-04-15T15:16:20.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0182967640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0182967640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c0d1b20 0x7f017c0c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0182967640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c1f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0182967640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f017c0d4f80 con 0x7f017c0b7be0 2024-04-15T15:16:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0182967640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f017c0d4240 con 0x7f017c0d1b20 2024-04-15T15:16:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0182967640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f017c0d5100 con 0x7f017c15e790 2024-04-15T15:16:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36808/0 (socket says 172.21.15.46:36808) 2024-04-15T15:16:20.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 -- 172.21.15.46:0/3556825886 learned_addr learned my addr 172.21.15.46:0/3556825886 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017b7fe640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c0d1b20 0x7f017c0c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f0180f04640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c1f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 -- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 msgr2=0x7f017c1f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c1f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 -- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c0d1b20 msgr2=0x7f017c0c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c0d1b20 0x7f017c0c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.023+0000 7f017bfff640 1 -- 172.21.15.46:0/3556825886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c1f4950 con 0x7f017c0b7be0 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.024+0000 7f017bfff640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c158b80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f016c00e870 tx=0x7f016c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.024+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c017070 con 0x7f017c0b7be0 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.024+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c00c940 con 0x7f017c0b7be0 2024-04-15T15:16:20.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.024+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f017c1f4c30 con 0x7f017c0b7be0 2024-04-15T15:16:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.024+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c00cd30 con 0x7f017c0b7be0 2024-04-15T15:16:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.024+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f017c0c8530 con 0x7f017c0b7be0 2024-04-15T15:16:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.025+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f017c0689e0 con 0x7f017c0b7be0 2024-04-15T15:16:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.025+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f016c01a070 con 0x7f017c0b7be0 2024-04-15T15:16:20.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.025+0000 7f0178ff9640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f016806c8f0 0x7f016806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.026+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f016c08bfd0 con 0x7f017c0b7be0 2024-04-15T15:16:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.026+0000 7f017b7fe640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f016806c8f0 0x7f016806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.026+0000 7f017b7fe640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f016806c8f0 0x7f016806edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0170035000 tx=0x7f0170008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.030+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f016c059a40 con 0x7f017c0b7be0 2024-04-15T15:16:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.216+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f017c0d0530 con 0x7f017c0b7be0 2024-04-15T15:16:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.218+0000 7f0178ff9640 1 -- 172.21.15.46:0/3556825886 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f016c05dd40 con 0x7f017c0b7be0 2024-04-15T15:16:20.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:20.220 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f016806c8f0 msgr2=0x7f016806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f016806c8f0 0x7f016806edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0170035000 tx=0x7f0170008030 comp rx=0 tx=0).stop 2024-04-15T15:16:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 msgr2=0x7f017c158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c158b80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f016c00e870 tx=0x7f016c00ed30 comp rx=0 tx=0).stop 2024-04-15T15:16:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 shutdown_connections 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f016806c8f0 0x7f016806edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f017c15e790 0x7f017c1f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c0d1b20 0x7f017c0c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 --2- 172.21.15.46:0/3556825886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f017c0b7be0 0x7f017c158b80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 >> 172.21.15.46:0/3556825886 conn(0x7f017c067b40 msgr2=0x7f017c0cb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.220+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 shutdown_connections 2024-04-15T15:16:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.221+0000 7f0182967640 1 -- 172.21.15.46:0/3556825886 wait complete. 2024-04-15T15:16:20.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:20.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.341+0000 7fa4bfcf2640 1 Processor -- start 2024-04-15T15:16:20.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.341+0000 7fa4bfcf2640 1 -- start start 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.342+0000 7fa4bfcf2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 0x7fa4b80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.342+0000 7fa4bfcf2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa4b80d2360 0x7fa4b80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.342+0000 7fa4bfcf2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4b80b47a0 con 0x7fa4b815eae0 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.342+0000 7fa4bfcf2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa4b80b53b0 con 0x7fa4b80d2360 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.342+0000 7fa4bfcf2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b80b4a80 con 0x7fa4b815e700 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bda8e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 0x7fa4b80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bd28d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa4b80d2360 0x7fa4b80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bda8e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 0x7fa4b80b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:36562/0 (socket says 172.21.15.46:36562) 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bda8e640 1 -- 172.21.15.46:0/1948627470 learned_addr learned my addr 172.21.15.46:0/1948627470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bd28d640 1 -- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 msgr2=0x7fa4b80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bd28d640 1 --2- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 0x7fa4b80b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bd28d640 1 -- 172.21.15.46:0/1948627470 >> v1:172.21.15.78:6789/0 conn(0x7fa4b815eae0 legacy=0x7fa4b80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bd28d640 1 -- 172.21.15.46:0/1948627470 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b80b4d60 con 0x7fa4b80d2360 2024-04-15T15:16:20.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bca8c640 1 -- 172.21.15.46:0/1948627470 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1809307281 0 0) 0x7fa4b80b47a0 con 0x7fa4b815eae0 2024-04-15T15:16:20.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.343+0000 7fa4bda8e640 1 --2- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 0x7fa4b80b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.344+0000 7fa4bd28d640 1 --2- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b80d2360 0x7fa4b80d2b60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0097a0 tx=0x7fa4ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6989fa7820e6f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.344+0000 7fa4bca8c640 1 -- 172.21.15.46:0/1948627470 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac035020 con 0x7fa4b80d2360 2024-04-15T15:16:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.344+0000 7fa4bca8c640 1 -- 172.21.15.46:0/1948627470 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ac031830 con 0x7fa4b80d2360 2024-04-15T15:16:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.344+0000 7fa4bca8c640 1 -- 172.21.15.46:0/1948627470 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac031c00 con 0x7fa4b80d2360 2024-04-15T15:16:20.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.344+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b80d2360 msgr2=0x7fa4b80d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.344+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b80d2360 0x7fa4b80d2b60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0097a0 tx=0x7fa4ac02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.345+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/1948627470 shutdown_connections 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.345+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b80d2360 0x7fa4b80d2b60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.345+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/1948627470 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b815e700 0x7fa4b80b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.345+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/1948627470 >> 172.21.15.46:0/1948627470 conn(0x7fa4b808bf60 msgr2=0x7fa4b80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.345+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/1948627470 shutdown_connections 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.345+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/1948627470 wait complete. 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.346+0000 7fa4bfcf2640 1 Processor -- start 2024-04-15T15:16:20.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.346+0000 7fa4bfcf2640 1 -- start start 2024-04-15T15:16:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.346+0000 7fa4bfcf2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b80d2360 0x7fa4b80cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.346+0000 7fa4bfcf2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 0x7fa4b8153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bfcf2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b815eae0 0x7fa4b80b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bfcf2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b80b5d80 con 0x7fa4b815e700 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bfcf2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b80b4450 con 0x7fa4b815eae0 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bfcf2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b80d1d40 con 0x7fa4b80d2360 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 0x7fa4b8153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 0x7fa4b8153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36816/0 (socket says 172.21.15.46:36816) 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 -- 172.21.15.46:0/3809229069 learned_addr learned my addr 172.21.15.46:0/3809229069 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bda8e640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b80d2360 0x7fa4b80cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4be28f640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b815eae0 0x7fa4b80b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 -- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b80d2360 msgr2=0x7fa4b80cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b80d2360 0x7fa4b80cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 -- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b815eae0 msgr2=0x7fa4b80b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b815eae0 0x7fa4b80b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 -- 172.21.15.46:0/3809229069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b80b87e0 con 0x7fa4b815e700 2024-04-15T15:16:20.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bd28d640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 0x7fa4b8153ba0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0097a0 tx=0x7fa4ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac035070 con 0x7fa4b815e700 2024-04-15T15:16:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ac004230 con 0x7fa4b815e700 2024-04-15T15:16:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b80b8a80 con 0x7fa4b815e700 2024-04-15T15:16:20.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac03a630 con 0x7fa4b815e700 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.347+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b80b9040 con 0x7fa4b815e700 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.348+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b80689e0 con 0x7fa4b815e700 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.349+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4ac038020 con 0x7fa4b815e700 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.349+0000 7fa4b67fc640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4a406c920 0x7fa4a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.349+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa4ac0ad9a0 con 0x7fa4b815e700 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.350+0000 7fa4bda8e640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4a406c920 0x7fa4a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.350+0000 7fa4bda8e640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4a406c920 0x7fa4a406ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009c90 tx=0x7fa4a800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.354+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4ac040070 con 0x7fa4b815e700 2024-04-15T15:16:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.538+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa4b80ba500 con 0x7fa4b815e700 2024-04-15T15:16:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.539+0000 7fa4b67fc640 1 -- 172.21.15.46:0/3809229069 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fa4ac07b410 con 0x7fa4b815e700 2024-04-15T15:16:20.542 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4a406c920 msgr2=0x7fa4a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4a406c920 0x7fa4a406ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009c90 tx=0x7fa4a800b040 comp rx=0 tx=0).stop 2024-04-15T15:16:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 msgr2=0x7fa4b8153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 0x7fa4b8153ba0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0097a0 tx=0x7fa4ac0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 shutdown_connections 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4a406c920 0x7fa4a406ede0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4b815eae0 0x7fa4b80b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4b815e700 0x7fa4b8153ba0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 --2- 172.21.15.46:0/3809229069 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4b80d2360 0x7fa4b80cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.540+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 >> 172.21.15.46:0/3809229069 conn(0x7fa4b808bf60 msgr2=0x7fa4b80cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.541+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 shutdown_connections 2024-04-15T15:16:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.541+0000 7fa4bfcf2640 1 -- 172.21.15.46:0/3809229069 wait complete. 2024-04-15T15:16:20.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:20.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 Processor -- start 2024-04-15T15:16:20.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 -- start start 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa4d015e9a0 0x7fa4d00d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4d00d1400 0x7fa4d00cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d00d1e60 con 0x7fa4d00cf750 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d00d18a0 con 0x7fa4d00d1400 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.660+0000 7fa4d704e640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d00d1b80 con 0x7fa4d015e9a0 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4d00d1400 0x7fa4d00cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4d00d1400 0x7fa4d00cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36824/0 (socket says 172.21.15.46:36824) 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 -- 172.21.15.46:0/1172635823 learned_addr learned my addr 172.21.15.46:0/1172635823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4d4dea640 1 --2- 172.21.15.46:0/1172635823 >> v2:172.21.15.78:3300/0 conn(0x7fa4d015e9a0 0x7fa4d00d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 -- 172.21.15.46:0/1172635823 >> v2:172.21.15.78:3300/0 conn(0x7fa4d015e9a0 msgr2=0x7fa4d00d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 --2- 172.21.15.46:0/1172635823 >> v2:172.21.15.78:3300/0 conn(0x7fa4d015e9a0 0x7fa4d00d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 -- 172.21.15.46:0/1172635823 >> v1:172.21.15.78:6789/0 conn(0x7fa4d00cf750 legacy=0x7fa4d00d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.661+0000 7fa4cffff640 1 -- 172.21.15.46:0/1172635823 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d006cff0 con 0x7fa4d00d1400 2024-04-15T15:16:20.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4cffff640 1 --2- 172.21.15.46:0/1172635823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d00d1400 0x7fa4d00cf210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa4d008dfc0 tx=0x7fa4bc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3c05b55e494e7f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4cf7fe640 1 -- 172.21.15.46:0/1172635823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc034070 con 0x7fa4d00d1400 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4cf7fe640 1 -- 172.21.15.46:0/1172635823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4bc030860 con 0x7fa4d00d1400 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4cf7fe640 1 -- 172.21.15.46:0/1172635823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc030c30 con 0x7fa4d00d1400 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 -- 172.21.15.46:0/1172635823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d00d1400 msgr2=0x7fa4d00cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 --2- 172.21.15.46:0/1172635823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d00d1400 0x7fa4d00cf210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa4d008dfc0 tx=0x7fa4bc02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 -- 172.21.15.46:0/1172635823 shutdown_connections 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 --2- 172.21.15.46:0/1172635823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d00d1400 0x7fa4d00cf210 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 --2- 172.21.15.46:0/1172635823 >> v2:172.21.15.78:3300/0 conn(0x7fa4d015e9a0 0x7fa4d00d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 -- 172.21.15.46:0/1172635823 >> 172.21.15.46:0/1172635823 conn(0x7fa4d009d950 msgr2=0x7fa4d008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 -- 172.21.15.46:0/1172635823 shutdown_connections 2024-04-15T15:16:20.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.662+0000 7fa4d704e640 1 -- 172.21.15.46:0/1172635823 wait complete. 2024-04-15T15:16:20.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 Processor -- start 2024-04-15T15:16:20.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 -- start start 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d00cf750 0x7fa4d0157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d00d1400 0x7fa4d01616d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 0x7fa4d015f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d00d24e0 con 0x7fa4d015e9a0 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d00d2360 con 0x7fa4d00d1400 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.664+0000 7fa4d704e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d00d32b0 con 0x7fa4d00cf750 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 0x7fa4d015f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 0x7fa4d015f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36836/0 (socket says 172.21.15.46:36836) 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 -- 172.21.15.46:0/69758359 learned_addr learned my addr 172.21.15.46:0/69758359 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4cffff640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d00d1400 0x7fa4d01616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d4dea640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d00cf750 0x7fa4d0157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 -- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d00cf750 msgr2=0x7fa4d0157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d00cf750 0x7fa4d0157e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 -- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d00d1400 msgr2=0x7fa4d01616d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d00d1400 0x7fa4d01616d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 -- 172.21.15.46:0/69758359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d00c9f40 con 0x7fa4d015e9a0 2024-04-15T15:16:20.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d55eb640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 0x7fa4d015f790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800c710 tx=0x7fa4c800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8016020 con 0x7fa4d015e9a0 2024-04-15T15:16:20.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c8006d50 con 0x7fa4d015e9a0 2024-04-15T15:16:20.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d00ca220 con 0x7fa4d015e9a0 2024-04-15T15:16:20.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8003bb0 con 0x7fa4d015e9a0 2024-04-15T15:16:20.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.665+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d00ca7e0 con 0x7fa4d015e9a0 2024-04-15T15:16:20.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.666+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d00690f0 con 0x7fa4d015e9a0 2024-04-15T15:16:20.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.667+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4c800a850 con 0x7fa4d015e9a0 2024-04-15T15:16:20.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.667+0000 7fa4cd7fa640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.667+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa4c808ba10 con 0x7fa4d015e9a0 2024-04-15T15:16:20.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.667+0000 7fa4d4dea640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.668+0000 7fa4d4dea640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00c80d0 tx=0x7fa4c4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.671+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4c8020020 con 0x7fa4d015e9a0 2024-04-15T15:16:20.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.857+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa4d015f570 con 0x7fa4d015e9a0 2024-04-15T15:16:20.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.858+0000 7fa4cd7fa640 1 -- 172.21.15.46:0/69758359 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa4c800ce20 con 0x7fa4d015e9a0 2024-04-15T15:16:20.860 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:20.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 msgr2=0x7fa4b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00c80d0 tx=0x7fa4c4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 msgr2=0x7fa4d015f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 0x7fa4d015f790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800c710 tx=0x7fa4c800cbd0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 shutdown_connections 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d015e9a0 0x7fa4d015f790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d00d1400 0x7fa4d01616d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 --2- 172.21.15.46:0/69758359 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d00cf750 0x7fa4d0157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 >> 172.21.15.46:0/69758359 conn(0x7fa4d009d950 msgr2=0x7fa4d006d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 shutdown_connections 2024-04-15T15:16:20.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.860+0000 7fa4d704e640 1 -- 172.21.15.46:0/69758359 wait complete. 2024-04-15T15:16:20.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.978+0000 7fb395a93640 1 Processor -- start 2024-04-15T15:16:20.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.978+0000 7fb395a93640 1 -- start start 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb395a93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb3900d2360 0x7fb3900d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb395a93640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb39015eae0 0x7fb3900d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb395a93640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3900b47a0 con 0x7fb39015e700 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb395a93640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb3900b53b0 con 0x7fb3900d2360 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb395a93640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb3900b4a80 con 0x7fb39015eae0 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb38effd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb3900d2360 0x7fb3900d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb38f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb39015e700 0x7fb3900b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:55850/0 (socket says 172.21.15.46:55850) 2024-04-15T15:16:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38f7fe640 1 -- 172.21.15.46:0/1418368957 learned_addr learned my addr 172.21.15.46:0/1418368957 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.979+0000 7fb38ffff640 1 --2- 172.21.15.46:0/1418368957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb39015eae0 0x7fb3900d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38ffff640 1 -- 172.21.15.46:0/1418368957 >> v2:172.21.15.78:3300/0 conn(0x7fb3900d2360 msgr2=0x7fb3900d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38ffff640 1 --2- 172.21.15.46:0/1418368957 >> v2:172.21.15.78:3300/0 conn(0x7fb3900d2360 0x7fb3900d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38ffff640 1 -- 172.21.15.46:0/1418368957 >> v1:172.21.15.78:6789/0 conn(0x7fb39015e700 legacy=0x7fb3900b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38e7fc640 1 -- 172.21.15.46:0/1418368957 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107699799 0 0) 0x7fb3900b47a0 con 0x7fb39015e700 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38ffff640 1 -- 172.21.15.46:0/1418368957 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900b4d60 con 0x7fb39015eae0 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.980+0000 7fb38effd640 1 --2- 172.21.15.46:0/1418368957 >> v2:172.21.15.78:3300/0 conn(0x7fb3900d2360 0x7fb3900d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb38ffff640 1 --2- 172.21.15.46:0/1418368957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb39015eae0 0x7fb3900d30a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb388009d90 tx=0x7fb38802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2075c268a486504d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb38e7fc640 1 -- 172.21.15.46:0/1418368957 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388039070 con 0x7fb39015eae0 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb38e7fc640 1 -- 172.21.15.46:0/1418368957 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38802cc50 con 0x7fb39015eae0 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb38e7fc640 1 -- 172.21.15.46:0/1418368957 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388034440 con 0x7fb39015eae0 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb395a93640 1 -- 172.21.15.46:0/1418368957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb39015eae0 msgr2=0x7fb3900d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb395a93640 1 --2- 172.21.15.46:0/1418368957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb39015eae0 0x7fb3900d30a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb388009d90 tx=0x7fb38802bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb395a93640 1 -- 172.21.15.46:0/1418368957 shutdown_connections 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb395a93640 1 --2- 172.21.15.46:0/1418368957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb39015eae0 0x7fb3900d30a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb395a93640 1 --2- 172.21.15.46:0/1418368957 >> v2:172.21.15.78:3300/0 conn(0x7fb3900d2360 0x7fb3900d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.981+0000 7fb395a93640 1 -- 172.21.15.46:0/1418368957 >> 172.21.15.46:0/1418368957 conn(0x7fb39008bf60 msgr2=0x7fb3900d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.982+0000 7fb395a93640 1 -- 172.21.15.46:0/1418368957 shutdown_connections 2024-04-15T15:16:20.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.982+0000 7fb395a93640 1 -- 172.21.15.46:0/1418368957 wait complete. 2024-04-15T15:16:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.983+0000 7fb395a93640 1 Processor -- start 2024-04-15T15:16:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.983+0000 7fb395a93640 1 -- start start 2024-04-15T15:16:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.983+0000 7fb395a93640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb3900d2360 0x7fb3900cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39015e700 0x7fb3901615a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 0x7fb39014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900d1d40 con 0x7fb39015eae0 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900b4450 con 0x7fb39015e700 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb3900d1ec0 con 0x7fb3900d2360 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 0x7fb39014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 0x7fb39014d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36848/0 (socket says 172.21.15.46:36848) 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 -- 172.21.15.46:0/2554141906 learned_addr learned my addr 172.21.15.46:0/2554141906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38f7fe640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb3900d2360 0x7fb3900cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38effd640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39015e700 0x7fb3901615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 -- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb3900d2360 msgr2=0x7fb3900cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb3900d2360 0x7fb3900cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 -- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39015e700 msgr2=0x7fb3901615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39015e700 0x7fb3901615a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 -- 172.21.15.46:0/2554141906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900caf30 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb38ffff640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 0x7fb39014d690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb388009d90 tx=0x7fb388004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388044020 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38803a040 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3900b9ad0 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb388039050 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.984+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3900b8ff0 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.986+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3900689e0 con 0x7fb39015eae0 2024-04-15T15:16:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.986+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb388003990 con 0x7fb39015eae0 2024-04-15T15:16:20.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.986+0000 7fb377fff640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb37c06c8f0 0x7fb37c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.986+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb3880ae190 con 0x7fb39015eae0 2024-04-15T15:16:20.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.986+0000 7fb38f7fe640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb37c06c8f0 0x7fb37c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.987+0000 7fb38f7fe640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb37c06c8f0 0x7fb37c06edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb380009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:20.991+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb38804e060 con 0x7fb39015eae0 2024-04-15T15:16:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.174+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb3900ba500 con 0x7fb39015eae0 2024-04-15T15:16:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.175+0000 7fb377fff640 1 -- 172.21.15.46:0/2554141906 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb38807bc00 con 0x7fb39015eae0 2024-04-15T15:16:21.177 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb37c06c8f0 msgr2=0x7fb37c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb37c06c8f0 0x7fb37c06edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb380007fd0 tx=0x7fb380009040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 msgr2=0x7fb39014d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 0x7fb39014d690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb388009d90 tx=0x7fb388004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 shutdown_connections 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb37c06c8f0 0x7fb37c06edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39015eae0 0x7fb39014d690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb39015e700 0x7fb3901615a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 --2- 172.21.15.46:0/2554141906 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb3900d2360 0x7fb3900cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 >> 172.21.15.46:0/2554141906 conn(0x7fb39008bf60 msgr2=0x7fb3900d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 shutdown_connections 2024-04-15T15:16:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.177+0000 7fb395a93640 1 -- 172.21.15.46:0/2554141906 wait complete. 2024-04-15T15:16:21.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.296+0000 7f2610877640 1 Processor -- start 2024-04-15T15:16:21.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.296+0000 7f2610877640 1 -- start start 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2610877640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c15e790 0x7f260c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2610877640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f260c0d1cf0 0x7f260c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2610877640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f260c15eb70 con 0x7f260c0b7be0 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2610877640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f260c0d3790 con 0x7f260c0d1cf0 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2610877640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f260c0d45b0 con 0x7f260c15e790 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2609d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f260c0b7be0 0x7f260c0d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:55858/0 (socket says 172.21.15.46:55858) 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f2609d9b640 1 -- 172.21.15.46:0/2024171498 learned_addr learned my addr 172.21.15.46:0/2024171498 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f260ad9d640 1 --2- 172.21.15.46:0/2024171498 >> v2:172.21.15.78:3300/0 conn(0x7f260c0d1cf0 0x7f260c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.297+0000 7f260a59c640 1 --2- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c15e790 0x7f260c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.298+0000 7f260ad9d640 1 -- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c15e790 msgr2=0x7f260c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.298+0000 7f260ad9d640 1 --2- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c15e790 0x7f260c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.298+0000 7f260ad9d640 1 -- 172.21.15.46:0/2024171498 >> v1:172.21.15.78:6789/0 conn(0x7f260c0b7be0 legacy=0x7f260c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.298+0000 7f260ad9d640 1 -- 172.21.15.46:0/2024171498 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260c0d3c80 con 0x7f260c0d1cf0 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f260ad9d640 1 --2- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0d1cf0 0x7f260c0d4a40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f260c0d3910 tx=0x7f260402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d99109c31bef4cac server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f260959a640 1 -- 172.21.15.46:0/2024171498 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604039070 con 0x7f260c0d1cf0 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f260959a640 1 -- 172.21.15.46:0/2024171498 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26040316f0 con 0x7f260c0d1cf0 2024-04-15T15:16:21.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f260959a640 1 -- 172.21.15.46:0/2024171498 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604031ac0 con 0x7f260c0d1cf0 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 -- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0d1cf0 msgr2=0x7f260c0d4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 --2- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0d1cf0 0x7f260c0d4a40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f260c0d3910 tx=0x7f260402b790 comp rx=0 tx=0).stop 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 -- 172.21.15.46:0/2024171498 shutdown_connections 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 --2- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0d1cf0 0x7f260c0d4a40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 --2- 172.21.15.46:0/2024171498 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c15e790 0x7f260c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 -- 172.21.15.46:0/2024171498 >> 172.21.15.46:0/2024171498 conn(0x7f260c067b40 msgr2=0x7f260c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 -- 172.21.15.46:0/2024171498 shutdown_connections 2024-04-15T15:16:21.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.299+0000 7f2610877640 1 -- 172.21.15.46:0/2024171498 wait complete. 2024-04-15T15:16:21.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.301+0000 7f2610877640 1 Processor -- start 2024-04-15T15:16:21.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.301+0000 7f2610877640 1 -- start start 2024-04-15T15:16:21.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.301+0000 7f2610877640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0b7be0 0x7f260c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2610877640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c0d1cf0 0x7f260c1f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2610877640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 0x7f260c1f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2610877640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f260c0d4f80 con 0x7f260c15e790 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2610877640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f260c0d4240 con 0x7f260c0b7be0 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2610877640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f260c0d5100 con 0x7f260c0d1cf0 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 0x7f260c1f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 0x7f260c1f80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36856/0 (socket says 172.21.15.46:36856) 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 -- 172.21.15.46:0/838938383 learned_addr learned my addr 172.21.15.46:0/838938383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2609d9b640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c0d1cf0 0x7f260c1f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260a59c640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0b7be0 0x7f260c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 -- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c0d1cf0 msgr2=0x7f260c1f4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c0d1cf0 0x7f260c1f4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 -- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0b7be0 msgr2=0x7f260c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0b7be0 0x7f260c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 -- 172.21.15.46:0/838938383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260c1f4820 con 0x7f260c15e790 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f260ad9d640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 0x7f260c1f80f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f260402b7d0 tx=0x7f26040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.302+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604043070 con 0x7f260c15e790 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.303+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26040346a0 con 0x7f260c15e790 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.303+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260c1f4b00 con 0x7f260c15e790 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.303+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604039040 con 0x7f260c15e790 2024-04-15T15:16:21.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.303+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260c0c9310 con 0x7f260c15e790 2024-04-15T15:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.304+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f260c0689e0 con 0x7f260c15e790 2024-04-15T15:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.304+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f260403c070 con 0x7f260c15e790 2024-04-15T15:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.305+0000 7f2602ffd640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25ec06c8f0 0x7f25ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.305+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f26040ad760 con 0x7f260c15e790 2024-04-15T15:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.305+0000 7f260a59c640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25ec06c8f0 0x7f25ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.305+0000 7f260a59c640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25ec06c8f0 0x7f25ec06edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f260c0b9d30 tx=0x7f25fc00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.309+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f260407b140 con 0x7f260c15e790 2024-04-15T15:16:21.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.492+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f260c1f5020 con 0x7f260c15e790 2024-04-15T15:16:21.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.493+0000 7f2602ffd640 1 -- 172.21.15.46:0/838938383 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f260407f440 con 0x7f260c15e790 2024-04-15T15:16:21.495 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:21.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25ec06c8f0 msgr2=0x7f25ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25ec06c8f0 0x7f25ec06edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f260c0b9d30 tx=0x7f25fc00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 msgr2=0x7f260c1f80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 0x7f260c1f80f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f260402b7d0 tx=0x7f26040046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 shutdown_connections 2024-04-15T15:16:21.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25ec06c8f0 0x7f25ec06edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f260c15e790 0x7f260c1f80f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f260c0d1cf0 0x7f260c1f4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 --2- 172.21.15.46:0/838938383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f260c0b7be0 0x7f260c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 >> 172.21.15.46:0/838938383 conn(0x7f260c067b40 msgr2=0x7f260c0b8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 shutdown_connections 2024-04-15T15:16:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.495+0000 7f2610877640 1 -- 172.21.15.46:0/838938383 wait complete. 2024-04-15T15:16:21.508 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 Processor -- start 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 -- start start 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f45a415e700 0x7f45a40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d3270 0x7f45a40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a40b47a0 con 0x7f45a40d2360 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f45a40b53b0 con 0x7f45a415e700 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.611+0000 7f45ab196640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f45a40b4a80 con 0x7f45a40d3270 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a8f32640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f45a415e700 0x7f45a40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a9733640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d3270 0x7f45a40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f45a40d2360 0x7f45a40d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:55870/0 (socket says 172.21.15.46:55870) 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a3fff640 1 -- 172.21.15.46:0/1680573199 learned_addr learned my addr 172.21.15.46:0/1680573199 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a8f32640 1 -- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d3270 msgr2=0x7f45a40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a8f32640 1 --2- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d3270 0x7f45a40cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a8f32640 1 -- 172.21.15.46:0/1680573199 >> v1:172.21.15.78:6789/0 conn(0x7f45a40d2360 legacy=0x7f45a40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.612+0000 7f45a8f32640 1 -- 172.21.15.46:0/1680573199 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a40b4d60 con 0x7f45a415e700 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.613+0000 7f45a8f32640 1 --2- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b5840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4598009790 tx=0x7f459802b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70303ee29e21d04b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.613+0000 7f45a37fe640 1 -- 172.21.15.46:0/1680573199 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598035020 con 0x7f45a415e700 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.613+0000 7f45a37fe640 1 -- 172.21.15.46:0/1680573199 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4598031800 con 0x7f45a415e700 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.613+0000 7f45a37fe640 1 -- 172.21.15.46:0/1680573199 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598031bd0 con 0x7f45a415e700 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 -- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 msgr2=0x7f45a40b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 --2- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b5840 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4598009790 tx=0x7f459802b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 -- 172.21.15.46:0/1680573199 shutdown_connections 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 --2- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d3270 0x7f45a40cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 --2- 172.21.15.46:0/1680573199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b5840 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 -- 172.21.15.46:0/1680573199 >> 172.21.15.46:0/1680573199 conn(0x7f45a408bf60 msgr2=0x7f45a40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 -- 172.21.15.46:0/1680573199 shutdown_connections 2024-04-15T15:16:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.614+0000 7f45ab196640 1 -- 172.21.15.46:0/1680573199 wait complete. 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.615+0000 7f45ab196640 1 Processor -- start 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.615+0000 7f45ab196640 1 -- start start 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45ab196640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d2360 0x7f45a4159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45ab196640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 0x7f45a40b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45ab196640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45ab196640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b5d80 con 0x7f45a40d3270 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45ab196640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40b4450 con 0x7f45a415e700 2024-04-15T15:16:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45ab196640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f45a40d1d40 con 0x7f45a40d2360 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 0x7f45a40b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 0x7f45a40b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36868/0 (socket says 172.21.15.46:36868) 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 -- 172.21.15.46:0/3239869057 learned_addr learned my addr 172.21.15.46:0/3239869057 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a9733640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a8f32640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d2360 0x7f45a4159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 -- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d2360 msgr2=0x7f45a4159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d2360 0x7f45a4159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 -- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 msgr2=0x7f45a40b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 -- 172.21.15.46:0/3239869057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a40b3200 con 0x7f45a40d3270 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a3fff640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 0x7f45a40b1c70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f459000c430 tx=0x7f459000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590017070 con 0x7f45a40d3270 2024-04-15T15:16:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.616+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4590005ce0 con 0x7f45a40d3270 2024-04-15T15:16:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.617+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a40b34e0 con 0x7f45a40d3270 2024-04-15T15:16:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.617+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45900072f0 con 0x7f45a40d3270 2024-04-15T15:16:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.617+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a40b1470 con 0x7f45a40d3270 2024-04-15T15:16:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.618+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a40689e0 con 0x7f45a40d3270 2024-04-15T15:16:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.618+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f459001a070 con 0x7f45a40d3270 2024-04-15T15:16:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.619+0000 7f45a17fa640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f458806c8f0 0x7f458806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.619+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f459008b2e0 con 0x7f45a40d3270 2024-04-15T15:16:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.619+0000 7f45a8f32640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f458806c8f0 0x7f458806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.619+0000 7f45a8f32640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f458806c8f0 0x7f458806edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4598036000 tx=0x7f459803c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.623+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45900593a0 con 0x7f45a40d3270 2024-04-15T15:16:21.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.810+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45a408e690 con 0x7f45a40d3270 2024-04-15T15:16:21.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.811+0000 7f45a17fa640 1 -- 172.21.15.46:0/3239869057 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f4590058ca0 con 0x7f45a40d3270 2024-04-15T15:16:21.813 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:21.813 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f458806c8f0 msgr2=0x7f458806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f458806c8f0 0x7f458806edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4598036000 tx=0x7f459803c040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 msgr2=0x7f45a40b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 0x7f45a40b1c70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f459000c430 tx=0x7f459000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 shutdown_connections 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f458806c8f0 0x7f458806edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a415e700 0x7f45a40b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45a40d3270 0x7f45a40b1c70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 --2- 172.21.15.46:0/3239869057 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45a40d2360 0x7f45a4159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 >> 172.21.15.46:0/3239869057 conn(0x7f45a408bf60 msgr2=0x7f45a40c8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 shutdown_connections 2024-04-15T15:16:21.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.813+0000 7f45ab196640 1 -- 172.21.15.46:0/3239869057 wait complete. 2024-04-15T15:16:21.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:21.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:21.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.932+0000 7fbfd5d00640 1 Processor -- start 2024-04-15T15:16:21.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.932+0000 7fbfd5d00640 1 -- start start 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfd5d00640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbfd015e790 0x7fbfd00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfd5d00640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 0x7fbfd00d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfd5d00640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd015eb70 con 0x7fbfd00b7be0 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfd5d00640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd00d3790 con 0x7fbfd015e790 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfd5d00640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd00d45b0 con 0x7fbfd00d1cf0 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfc6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbfd00b7be0 0x7fbfd00d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34862/0 (socket says 172.21.15.46:34862) 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfc6ffd640 1 -- 172.21.15.46:0/1118813819 learned_addr learned my addr 172.21.15.46:0/1118813819 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/1118813819 >> v2:172.21.15.78:3300/0 conn(0x7fbfd015e790 0x7fbfd00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfcffff640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 0x7fbfd00d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1272272953 0 0) 0x7fbfd015eb70 con 0x7fbfd00b7be0 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfbc003520 con 0x7fbfd00b7be0 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 64159392 0 0) 0x7fbfbc003520 con 0x7fbfd00b7be0 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 msgr2=0x7fbfd00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 0x7fbfd00d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 >> v2:172.21.15.78:3300/0 conn(0x7fbfd015e790 msgr2=0x7fbfd00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 --2- 172.21.15.46:0/1118813819 >> v2:172.21.15.78:3300/0 conn(0x7fbfd015e790 0x7fbfd00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd00d3c80 con 0x7fbfd00b7be0 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fbfb8001690 con 0x7fbfd00b7be0 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 >> v1:172.21.15.46:6789/0 conn(0x7fbfd00b7be0 legacy=0x7fbfd00d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfbc0039b0 0x7fbfbc003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfbc004800 0x7fbfbc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 0x7fbfbc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd00d3790 con 0x7fbfbc0039b0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd00d45b0 con 0x7fbfbc004800 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.933+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbfbc00baf0 con 0x7fbfd00d1cf0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfbc0039b0 0x7fbfbc003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfc6ffd640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfbc004800 0x7fbfbc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcffff640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 0x7fbfbc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcf7fe640 1 -- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 msgr2=0x7fbfbc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfd00d1cf0 0x7fbfbc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcf7fe640 1 -- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfbc004800 msgr2=0x7fbfbc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfbc004800 0x7fbfbc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.934+0000 7fbfcf7fe640 1 -- 172.21.15.46:0/1118813819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfbc003520 con 0x7fbfbc0039b0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfbc0039b0 0x7fbfbc003df0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009a90 tx=0x7fbfbc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a390ba2a572d734 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0039070 con 0x7fbfbc0039b0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc002c6b0 con 0x7fbfbc0039b0 2024-04-15T15:16:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfceffd640 1 -- 172.21.15.46:0/1118813819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0009690 con 0x7fbfbc0039b0 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfd5d00640 1 -- 172.21.15.46:0/1118813819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfbc0039b0 msgr2=0x7fbfbc003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/1118813819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfbc0039b0 0x7fbfbc003df0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009a90 tx=0x7fbfbc005830 comp rx=0 tx=0).stop 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfcffff640 1 -- 172.21.15.46:0/1118813819 reap_dead start 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfd5d00640 1 -- 172.21.15.46:0/1118813819 shutdown_connections 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfd5d00640 1 -- 172.21.15.46:0/1118813819 >> 172.21.15.46:0/1118813819 conn(0x7fbfd0067b40 msgr2=0x7fbfd0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfd5d00640 1 -- 172.21.15.46:0/1118813819 shutdown_connections 2024-04-15T15:16:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.935+0000 7fbfd5d00640 1 -- 172.21.15.46:0/1118813819 wait complete. 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.936+0000 7fbfd5d00640 1 Processor -- start 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 -- start start 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfbc0039b0 0x7fbfd01f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 0x7fbfd00c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfd00d28c0 0x7fbfd00d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd00d4a40 con 0x7fbfd00c9170 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd00d4240 con 0x7fbfd00d28c0 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfd5d00640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd00d4bc0 con 0x7fbfbc0039b0 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfc6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 0x7fbfd00c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfc6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 0x7fbfd00c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36894/0 (socket says 172.21.15.46:36894) 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.937+0000 7fbfc6ffd640 1 -- 172.21.15.46:0/3587457212 learned_addr learned my addr 172.21.15.46:0/3587457212 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:21.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfcffff640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfd00d28c0 0x7fbfd00d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfbc0039b0 0x7fbfd01f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfc6ffd640 1 -- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfbc0039b0 msgr2=0x7fbfd01f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfc6ffd640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfbc0039b0 0x7fbfd01f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfc6ffd640 1 -- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfd00d28c0 msgr2=0x7fbfd00d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfc6ffd640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfd00d28c0 0x7fbfd00d2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfc6ffd640 1 -- 172.21.15.46:0/3587457212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd00c9d30 con 0x7fbfd00c9170 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfbc0039b0 0x7fbfd01f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfc6ffd640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 0x7fbfd00c95d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8007340 tx=0x7fbfb8007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8007e20 con 0x7fbfd00c9170 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb8018070 con 0x7fbfd00c9170 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd00ca010 con 0x7fbfd00c9170 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb800fb80 con 0x7fbfd00c9170 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.938+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd0159460 con 0x7fbfd00c9170 2024-04-15T15:16:21.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.939+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd00689e0 con 0x7fbfd00c9170 2024-04-15T15:16:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.939+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfb800fd20 con 0x7fbfd00c9170 2024-04-15T15:16:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.940+0000 7fbfccff9640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbfb006c8f0 0x7fbfb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.940+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbfb808bea0 con 0x7fbfd00c9170 2024-04-15T15:16:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.940+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbfb006c8f0 0x7fbfb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.940+0000 7fbfcf7fe640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbfb006c8f0 0x7fbfb006edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0036000 tx=0x7fbfc0030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:21.944+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfb8059f60 con 0x7fbfd00c9170 2024-04-15T15:16:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.127+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbfd00d5250 con 0x7fbfd00c9170 2024-04-15T15:16:22.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.127+0000 7fbfccff9640 1 -- 172.21.15.46:0/3587457212 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fbfb8059860 con 0x7fbfd00c9170 2024-04-15T15:16:22.129 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.129+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbfb006c8f0 msgr2=0x7fbfb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.129+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbfb006c8f0 0x7fbfb006edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0036000 tx=0x7fbfc0030000 comp rx=0 tx=0).stop 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.129+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 msgr2=0x7fbfd00c95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.129+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 0x7fbfd00c95d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8007340 tx=0x7fbfb8007800 comp rx=0 tx=0).stop 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.129+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 shutdown_connections 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbfb006c8f0 0x7fbfb006edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbfd00d28c0 0x7fbfd00d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfd00c9170 0x7fbfd00c95d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 --2- 172.21.15.46:0/3587457212 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbfbc0039b0 0x7fbfd01f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 >> 172.21.15.46:0/3587457212 conn(0x7fbfd0067b40 msgr2=0x7fbfd00d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 shutdown_connections 2024-04-15T15:16:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.130+0000 7fbfd5d00640 1 -- 172.21.15.46:0/3587457212 wait complete. 2024-04-15T15:16:22.143 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:22.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.249+0000 7ffb745a8640 1 Processor -- start 2024-04-15T15:16:22.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.249+0000 7ffb745a8640 1 -- start start 2024-04-15T15:16:22.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb745a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ffb6c15e7a0 0x7ffb6c0d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb745a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6c0d0530 con 0x7ffb6c0d1870 2024-04-15T15:16:22.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb745a8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6c163190 con 0x7ffb6c0d13f0 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb745a8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6c0cf490 con 0x7ffb6c15e7a0 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb72b45640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ffb6c0d1870 0x7ffb6c162a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34878/0 (socket says 172.21.15.46:34878) 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb72b45640 1 -- 172.21.15.46:0/3127335325 learned_addr learned my addr 172.21.15.46:0/3127335325 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb72344640 1 --2- 172.21.15.46:0/3127335325 >> v2:172.21.15.78:3300/0 conn(0x7ffb6c15e7a0 0x7ffb6c0d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482666873 0 0) 0x7ffb6c0d0530 con 0x7ffb6c0d1870 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb58003520 con 0x7ffb6c0d1870 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524986579 0 0) 0x7ffb6c163190 con 0x7ffb6c0d13f0 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb6c0d0530 con 0x7ffb6c0d13f0 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1002055769 0 0) 0x7ffb58003520 con 0x7ffb6c0d1870 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb72344640 1 -- 172.21.15.46:0/3127335325 >> v1:172.21.15.78:6789/0 conn(0x7ffb6c0d13f0 legacy=0x7ffb6c162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb72344640 1 -- 172.21.15.46:0/3127335325 >> v1:172.21.15.46:6789/0 conn(0x7ffb6c0d1870 legacy=0x7ffb6c162a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.250+0000 7ffb72344640 1 -- 172.21.15.46:0/3127335325 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c0cf770 con 0x7ffb6c15e7a0 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.251+0000 7ffb72344640 1 --2- 172.21.15.46:0/3127335325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c0d0eb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb680097a0 tx=0x7ffb6802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69271da1e97a2aef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.251+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68037070 con 0x7ffb6c15e7a0 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb68030510 con 0x7ffb6c15e7a0 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb71342640 1 -- 172.21.15.46:0/3127335325 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb640019f0 con 0x7ffb6c15e7a0 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 -- 172.21.15.46:0/3127335325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 msgr2=0x7ffb6c0d0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 --2- 172.21.15.46:0/3127335325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c0d0eb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb680097a0 tx=0x7ffb6802b790 comp rx=0 tx=0).stop 2024-04-15T15:16:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 -- 172.21.15.46:0/3127335325 shutdown_connections 2024-04-15T15:16:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 --2- 172.21.15.46:0/3127335325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c0d0eb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 -- 172.21.15.46:0/3127335325 >> 172.21.15.46:0/3127335325 conn(0x7ffb6c08bf60 msgr2=0x7ffb6c0d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 -- 172.21.15.46:0/3127335325 shutdown_connections 2024-04-15T15:16:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.252+0000 7ffb745a8640 1 -- 172.21.15.46:0/3127335325 wait complete. 2024-04-15T15:16:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.253+0000 7ffb745a8640 1 Processor -- start 2024-04-15T15:16:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.253+0000 7ffb745a8640 1 -- start start 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb745a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ffb6c0d13f0 0x7ffb6c0ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb745a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 0x7ffb6c0c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb745a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c14cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb745a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c0cf180 con 0x7ffb6c0d1870 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb745a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6c0cfd30 con 0x7ffb6c15e7a0 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb745a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6c06db20 con 0x7ffb6c0d13f0 2024-04-15T15:16:22.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 0x7ffb6c0c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 0x7ffb6c0c7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36896/0 (socket says 172.21.15.46:36896) 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 -- 172.21.15.46:0/2845205423 learned_addr learned my addr 172.21.15.46:0/2845205423 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb72b45640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c14cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb72344640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ffb6c0d13f0 0x7ffb6c0ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 -- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ffb6c0d13f0 msgr2=0x7ffb6c0ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ffb6c0d13f0 0x7ffb6c0ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 -- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 msgr2=0x7ffb6c14cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c14cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.254+0000 7ffb71b43640 1 -- 172.21.15.46:0/2845205423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6c155e10 con 0x7ffb6c0d1870 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.255+0000 7ffb71b43640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 0x7ffb6c0c7db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00c430 tx=0x7ffb5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.255+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c017070 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.255+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5c0052f0 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.255+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6c0cd040 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.255+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c003830 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.255+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6c0cd8c0 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.256+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb6c0689e0 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.256+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffb5c01a070 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.257+0000 7ffb62ffd640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ffb5006c8f0 0x7ffb5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.257+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffb5c08b280 con 0x7ffb6c0d1870 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.257+0000 7ffb72344640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ffb5006c8f0 0x7ffb5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.258+0000 7ffb72344640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ffb5006c8f0 0x7ffb5006edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffb68034000 tx=0x7ffb6802f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.261+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb6c155e10 con 0x7ffb6c0d1870 2024-04-15T15:16:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.445+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb6c0d1d60 con 0x7ffb6c0d1870 2024-04-15T15:16:22.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.446+0000 7ffb62ffd640 1 -- 172.21.15.46:0/2845205423 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7ffb5c058c60 con 0x7ffb6c0d1870 2024-04-15T15:16:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ffb5006c8f0 msgr2=0x7ffb5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ffb5006c8f0 0x7ffb5006edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ffb68034000 tx=0x7ffb6802f000 comp rx=0 tx=0).stop 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 msgr2=0x7ffb6c0c7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 0x7ffb6c0c7db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00c430 tx=0x7ffb5c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 shutdown_connections 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ffb5006c8f0 0x7ffb5006edb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb6c15e7a0 0x7ffb6c14cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb6c0d1870 0x7ffb6c0c7db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 --2- 172.21.15.46:0/2845205423 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ffb6c0d13f0 0x7ffb6c0ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 >> 172.21.15.46:0/2845205423 conn(0x7ffb6c08bf60 msgr2=0x7ffb6c15eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 shutdown_connections 2024-04-15T15:16:22.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.448+0000 7ffb745a8640 1 -- 172.21.15.46:0/2845205423 wait complete. 2024-04-15T15:16:22.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:22.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.567+0000 7fb01378f640 1 Processor -- start 2024-04-15T15:16:22.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.567+0000 7fb01378f640 1 -- start start 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01378f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb00c15ec10 0x7fb00c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01378f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb00c0b53c0 0x7fb00c0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01378f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb00c067e30 con 0x7fb00c15e830 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01378f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb00c0b3c10 con 0x7fb00c15ec10 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01378f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb00c0b3ef0 con 0x7fb00c0b53c0 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb010d2a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb00c15ec10 0x7fb00c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01152b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb00c15e830 0x7fb00c067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34886/0 (socket says 172.21.15.46:34886) 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb01152b640 1 -- 172.21.15.46:0/1149657781 learned_addr learned my addr 172.21.15.46:0/1149657781 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:22.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb011d2c640 1 --2- 172.21.15.46:0/1149657781 >> v2:172.21.15.78:3300/0 conn(0x7fb00c0b53c0 0x7fb00c0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb010d2a640 1 -- 172.21.15.46:0/1149657781 >> v2:172.21.15.78:3300/0 conn(0x7fb00c0b53c0 msgr2=0x7fb00c0b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:22.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb010d2a640 1 --2- 172.21.15.46:0/1149657781 >> v2:172.21.15.78:3300/0 conn(0x7fb00c0b53c0 0x7fb00c0b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb010d2a640 1 -- 172.21.15.46:0/1149657781 >> v1:172.21.15.46:6789/0 conn(0x7fb00c15e830 legacy=0x7fb00c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb010d2a640 1 -- 172.21.15.46:0/1149657781 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c0b41d0 con 0x7fb00c15ec10 2024-04-15T15:16:22.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb003fff640 1 -- 172.21.15.46:0/1149657781 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839434391 0 0) 0x7fb00c067e30 con 0x7fb00c15e830 2024-04-15T15:16:22.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.568+0000 7fb011d2c640 1 --2- 172.21.15.46:0/1149657781 >> v2:172.21.15.78:3300/0 conn(0x7fb00c0b53c0 0x7fb00c0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.569+0000 7fb010d2a640 1 --2- 172.21.15.46:0/1149657781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c15ec10 0x7fb00c0b4cb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0b51f0 tx=0x7faffc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c8ebec46d1529c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.569+0000 7fb003fff640 1 -- 172.21.15.46:0/1149657781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc0315a0 con 0x7fb00c15ec10 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.569+0000 7fb003fff640 1 -- 172.21.15.46:0/1149657781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faffc037070 con 0x7fb00c15ec10 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.569+0000 7fb003fff640 1 -- 172.21.15.46:0/1149657781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc031960 con 0x7fb00c15ec10 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 -- 172.21.15.46:0/1149657781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c15ec10 msgr2=0x7fb00c0b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 --2- 172.21.15.46:0/1149657781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c15ec10 0x7fb00c0b4cb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0b51f0 tx=0x7faffc02b840 comp rx=0 tx=0).stop 2024-04-15T15:16:22.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 -- 172.21.15.46:0/1149657781 shutdown_connections 2024-04-15T15:16:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 --2- 172.21.15.46:0/1149657781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c15ec10 0x7fb00c0b4cb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 --2- 172.21.15.46:0/1149657781 >> v2:172.21.15.78:3300/0 conn(0x7fb00c0b53c0 0x7fb00c0b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 -- 172.21.15.46:0/1149657781 >> 172.21.15.46:0/1149657781 conn(0x7fb00c073360 msgr2=0x7fb00c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 -- 172.21.15.46:0/1149657781 shutdown_connections 2024-04-15T15:16:22.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.570+0000 7fb01378f640 1 -- 172.21.15.46:0/1149657781 wait complete. 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.571+0000 7fb01378f640 1 Processor -- start 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.571+0000 7fb01378f640 1 -- start start 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb00c0b53c0 0x7fb00c159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb00c15e830 0x7fb00c1e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 0x7fb00c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c15ff60 con 0x7fb00c0c5f10 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c0b4790 con 0x7fb00c15e830 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb00c1600e0 con 0x7fb00c0b53c0 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 0x7fb00c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 0x7fb00c0c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36918/0 (socket says 172.21.15.46:36918) 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 -- 172.21.15.46:0/2486505856 learned_addr learned my addr 172.21.15.46:0/2486505856 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:22.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb010d2a640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb00c15e830 0x7fb00c1e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01152b640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb00c0b53c0 0x7fb00c159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 -- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb00c0b53c0 msgr2=0x7fb00c159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb00c0b53c0 0x7fb00c159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 -- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb00c15e830 msgr2=0x7fb00c1e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb00c15e830 0x7fb00c1e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 -- 172.21.15.46:0/2486505856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c1e8f50 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb011d2c640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 0x7fb00c0c6470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb004007ef0 tx=0x7fb00400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00401a070 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb004003b90 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00c1e7eb0 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.572+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004004030 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.573+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00c1e8470 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.574+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00c159c70 con 0x7fb00c0c5f10 2024-04-15T15:16:22.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.574+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb00401d070 con 0x7fb00c0c5f10 2024-04-15T15:16:22.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.575+0000 7fb001ffb640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faff006c8f0 0x7faff006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.575+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb00408c0c0 con 0x7fb00c0c5f10 2024-04-15T15:16:22.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.575+0000 7fb01152b640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faff006c8f0 0x7faff006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.576+0000 7fb01152b640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faff006c8f0 0x7faff006edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb008000f60 tx=0x7fb008007d50 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.579+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb00c159c70 con 0x7fb00c0c5f10 2024-04-15T15:16:22.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.769+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb00c0689e0 con 0x7fb00c0c5f10 2024-04-15T15:16:22.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.769+0000 7fb001ffb640 1 -- 172.21.15.46:0/2486505856 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb004010070 con 0x7fb00c0c5f10 2024-04-15T15:16:22.772 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:22.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faff006c8f0 msgr2=0x7faff006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faff006c8f0 0x7faff006edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb008000f60 tx=0x7fb008007d50 comp rx=0 tx=0).stop 2024-04-15T15:16:22.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 msgr2=0x7fb00c0c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 0x7fb00c0c6470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb004007ef0 tx=0x7fb00400d4a0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 shutdown_connections 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7faff006c8f0 0x7faff006edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb00c0c5f10 0x7fb00c0c6470 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb00c15e830 0x7fb00c1e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 --2- 172.21.15.46:0/2486505856 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb00c0b53c0 0x7fb00c159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.771+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 >> 172.21.15.46:0/2486505856 conn(0x7fb00c073360 msgr2=0x7fb00c0af350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.772+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 shutdown_connections 2024-04-15T15:16:22.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.772+0000 7fb01378f640 1 -- 172.21.15.46:0/2486505856 wait complete. 2024-04-15T15:16:22.784 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.890+0000 7fcd81950640 1 Processor -- start 2024-04-15T15:16:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.890+0000 7fcd81950640 1 -- start start 2024-04-15T15:16:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.890+0000 7fcd81950640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd7c0b5230 0x7fcd7c072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.890+0000 7fcd81950640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd7c073080 con 0x7fcd7c158770 2024-04-15T15:16:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.890+0000 7fcd81950640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd7c0b4190 con 0x7fcd7c072c60 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.890+0000 7fcd81950640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcd7c0b4da0 con 0x7fcd7c0b5230 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.891+0000 7fcd7a7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd7c0b5230 0x7fcd7c072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.891+0000 7fcd7affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd7c158770 0x7fcd7c15bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34888/0 (socket says 172.21.15.46:34888) 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.891+0000 7fcd7affd640 1 -- 172.21.15.46:0/2399035815 learned_addr learned my addr 172.21.15.46:0/2399035815 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.891+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2399035815 >> v1:172.21.15.78:6789/0 conn(0x7fcd7c072c60 legacy=0x7fcd7c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.891+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2399035815 >> v1:172.21.15.46:6789/0 conn(0x7fcd7c158770 legacy=0x7fcd7c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.891+0000 7fcd7a7fc640 1 -- 172.21.15.46:0/2399035815 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c0b4470 con 0x7fcd7c0b5230 2024-04-15T15:16:22.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/2399035815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0b5230 0x7fcd7c072720 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcd700097a0 tx=0x7fcd7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e91bd3469fa1de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd79ffb640 1 -- 172.21.15.46:0/2399035815 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70035020 con 0x7fcd7c0b5230 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd79ffb640 1 -- 172.21.15.46:0/2399035815 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70031830 con 0x7fcd7c0b5230 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd79ffb640 1 -- 172.21.15.46:0/2399035815 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70031c00 con 0x7fcd7c0b5230 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 -- 172.21.15.46:0/2399035815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0b5230 msgr2=0x7fcd7c072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 --2- 172.21.15.46:0/2399035815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0b5230 0x7fcd7c072720 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcd700097a0 tx=0x7fcd7002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 -- 172.21.15.46:0/2399035815 shutdown_connections 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 --2- 172.21.15.46:0/2399035815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c0b5230 0x7fcd7c072720 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 -- 172.21.15.46:0/2399035815 >> 172.21.15.46:0/2399035815 conn(0x7fcd7c06be00 msgr2=0x7fcd7c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 -- 172.21.15.46:0/2399035815 shutdown_connections 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.892+0000 7fcd81950640 1 -- 172.21.15.46:0/2399035815 wait complete. 2024-04-15T15:16:22.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.894+0000 7fcd81950640 1 Processor -- start 2024-04-15T15:16:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.894+0000 7fcd81950640 1 -- start start 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.894+0000 7fcd81950640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 0x7fcd7c0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.894+0000 7fcd81950640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd7c0b5230 0x7fcd7c0ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.894+0000 7fcd81950640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd7c158770 0x7fcd7c1e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.894+0000 7fcd81950640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c071fa0 con 0x7fcd7c072c60 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd81950640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7c0b4a30 con 0x7fcd7c158770 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd81950640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7c158b50 con 0x7fcd7c0b5230 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 0x7fcd7c0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 0x7fcd7c0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36936/0 (socket says 172.21.15.46:36936) 2024-04-15T15:16:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 -- 172.21.15.46:0/3471234011 learned_addr learned my addr 172.21.15.46:0/3471234011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 -- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd7c0b5230 msgr2=0x7fcd7c0ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd7c0b5230 0x7fcd7c0ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 -- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd7c158770 msgr2=0x7fcd7c1e49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd7c158770 0x7fcd7c1e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 -- 172.21.15.46:0/3471234011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c15da50 con 0x7fcd7c072c60 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd7affd640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 0x7fcd7c0ce990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00c430 tx=0x7fcd6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c017070 con 0x7fcd7c072c60 2024-04-15T15:16:22.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd6c005ce0 con 0x7fcd7c072c60 2024-04-15T15:16:22.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7c15dd30 con 0x7fcd7c072c60 2024-04-15T15:16:22.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0072f0 con 0x7fcd7c072c60 2024-04-15T15:16:22.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.895+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd7c0cc7e0 con 0x7fcd7c072c60 2024-04-15T15:16:22.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.896+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7c0cc5f0 con 0x7fcd7c072c60 2024-04-15T15:16:22.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.897+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd6c01a070 con 0x7fcd7c072c60 2024-04-15T15:16:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.897+0000 7fcd63fff640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd6806c920 0x7fcd6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.897+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcd6c08b2e0 con 0x7fcd7c072c60 2024-04-15T15:16:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.897+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd6806c920 0x7fcd6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.898+0000 7fcd7a7fc640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd6806c920 0x7fcd6806ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcd70036000 tx=0x7fcd70030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:22.901+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd6c0593a0 con 0x7fcd7c072c60 2024-04-15T15:16:23.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.088+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcd7c0689e0 con 0x7fcd7c072c60 2024-04-15T15:16:23.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.088+0000 7fcd63fff640 1 -- 172.21.15.46:0/3471234011 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fcd6c058ca0 con 0x7fcd7c072c60 2024-04-15T15:16:23.090 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.090+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd6806c920 msgr2=0x7fcd6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd6806c920 0x7fcd6806ede0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcd70036000 tx=0x7fcd70030000 comp rx=0 tx=0).stop 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 msgr2=0x7fcd7c0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 0x7fcd7c0ce990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c00c430 tx=0x7fcd6c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 shutdown_connections 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd6806c920 0x7fcd6806ede0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd7c158770 0x7fcd7c1e49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd7c0b5230 0x7fcd7c0ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 --2- 172.21.15.46:0/3471234011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd7c072c60 0x7fcd7c0ce990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 >> 172.21.15.46:0/3471234011 conn(0x7fcd7c06be00 msgr2=0x7fcd7c0afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 shutdown_connections 2024-04-15T15:16:23.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.091+0000 7fcd81950640 1 -- 172.21.15.46:0/3471234011 wait complete. 2024-04-15T15:16:23.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.207+0000 7f0fef8eb640 1 Processor -- start 2024-04-15T15:16:23.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.207+0000 7f0fef8eb640 1 -- start start 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fef8eb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe8070940 0x7f0fe809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fef8eb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fe8160310 0x7f0fe8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fef8eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0fe80d0da0 0x7f0fe80d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fef8eb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe80d05e0 con 0x7f0fe8160310 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fef8eb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe8071260 con 0x7f0fe80d0da0 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fef8eb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe815f720 con 0x7f0fe8070940 2024-04-15T15:16:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fe8160310 0x7f0fe8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fe8160310 0x7f0fe8070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36944/0 (socket says 172.21.15.46:36944) 2024-04-15T15:16:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 -- 172.21.15.46:0/3377380101 learned_addr learned my addr 172.21.15.46:0/3377380101 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fede88640 1 --2- 172.21.15.46:0/3377380101 >> v2:172.21.15.78:3300/0 conn(0x7f0fe80d0da0 0x7f0fe80d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fed687640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe8070940 0x7f0fe809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 -- 172.21.15.46:0/3377380101 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe8070940 msgr2=0x7f0fe809b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe8070940 0x7f0fe809b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 -- 172.21.15.46:0/3377380101 >> v2:172.21.15.78:3300/0 conn(0x7f0fe80d0da0 msgr2=0x7f0fe80d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 --2- 172.21.15.46:0/3377380101 >> v2:172.21.15.78:3300/0 conn(0x7f0fe80d0da0 0x7f0fe80d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fece86640 1 -- 172.21.15.46:0/3377380101 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe815fa00 con 0x7f0fe8160310 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fede88640 1 --2- 172.21.15.46:0/3377380101 >> v2:172.21.15.78:3300/0 conn(0x7f0fe80d0da0 0x7f0fe80d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.208+0000 7f0fed687640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe8070940 0x7f0fe809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.209+0000 7f0fece86640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe8070d20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8073e20 tx=0x7f0fd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d495d99221c1192 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.209+0000 7f0fdffff640 1 -- 172.21.15.46:0/3377380101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8035020 con 0x7f0fe8160310 2024-04-15T15:16:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.209+0000 7f0fdffff640 1 -- 172.21.15.46:0/3377380101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fd8031870 con 0x7f0fe8160310 2024-04-15T15:16:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.209+0000 7f0fdffff640 1 -- 172.21.15.46:0/3377380101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8031c40 con 0x7f0fe8160310 2024-04-15T15:16:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 -- 172.21.15.46:0/3377380101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 msgr2=0x7f0fe8070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe8070d20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8073e20 tx=0x7f0fd802b910 comp rx=0 tx=0).stop 2024-04-15T15:16:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 -- 172.21.15.46:0/3377380101 shutdown_connections 2024-04-15T15:16:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe8070d20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/3377380101 >> v2:172.21.15.78:3300/0 conn(0x7f0fe80d0da0 0x7f0fe80d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/3377380101 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe8070940 0x7f0fe809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 -- 172.21.15.46:0/3377380101 >> 172.21.15.46:0/3377380101 conn(0x7f0fe80b3f40 msgr2=0x7f0fe80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 -- 172.21.15.46:0/3377380101 shutdown_connections 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.210+0000 7f0fef8eb640 1 -- 172.21.15.46:0/3377380101 wait complete. 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.211+0000 7f0fef8eb640 1 Processor -- start 2024-04-15T15:16:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.211+0000 7f0fef8eb640 1 -- start start 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe8070940 0x7f0fe80b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe80d0da0 0x7f0fe80c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe80c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe8160770 con 0x7f0fe8160310 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe80d0880 con 0x7f0fe8070940 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe80d1770 con 0x7f0fe80d0da0 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe80c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe80c5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36948/0 (socket says 172.21.15.46:36948) 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 -- 172.21.15.46:0/791943824 learned_addr learned my addr 172.21.15.46:0/791943824 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fed687640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe8070940 0x7f0fe80b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fece86640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe80d0da0 0x7f0fe80c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 -- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe80d0da0 msgr2=0x7f0fe80c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe80d0da0 0x7f0fe80c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 -- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe8070940 msgr2=0x7f0fe80b7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe8070940 0x7f0fe80b7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 -- 172.21.15.46:0/791943824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe8157040 con 0x7f0fe8160310 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fede88640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe80c5e30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000cf20 tx=0x7f0fe000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe001a070 con 0x7f0fe8160310 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe000d840 con 0x7f0fe8160310 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe815d5d0 con 0x7f0fe8160310 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.212+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe000dc30 con 0x7f0fe8160310 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.213+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe80b5d50 con 0x7f0fe8160310 2024-04-15T15:16:23.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.214+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe80689e0 con 0x7f0fe8160310 2024-04-15T15:16:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.214+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0fe001d070 con 0x7f0fe8160310 2024-04-15T15:16:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.214+0000 7f0fddffb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0fcc06c8f0 0x7f0fcc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.214+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0fe008c6d0 con 0x7f0fe8160310 2024-04-15T15:16:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.215+0000 7f0fed687640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0fcc06c8f0 0x7f0fcc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.215+0000 7f0fed687640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0fcc06c8f0 0x7f0fcc06edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4006b10 tx=0x7f0fe4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.219+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0fe005a140 con 0x7f0fe8160310 2024-04-15T15:16:23.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.407+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0fe8070720 con 0x7f0fe8160310 2024-04-15T15:16:23.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.409+0000 7f0fddffb640 1 -- 172.21.15.46:0/791943824 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f0fe005e440 con 0x7f0fe8160310 2024-04-15T15:16:23.411 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:23.411 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0fcc06c8f0 msgr2=0x7f0fcc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0fcc06c8f0 0x7f0fcc06edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4006b10 tx=0x7f0fe4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 msgr2=0x7f0fe80c5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe80c5e30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000cf20 tx=0x7f0fe000a7d0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 shutdown_connections 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0fcc06c8f0 0x7f0fcc06edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fe8160310 0x7f0fe80c5e30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0fe80d0da0 0x7f0fe80c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 --2- 172.21.15.46:0/791943824 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0fe8070940 0x7f0fe80b7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 >> 172.21.15.46:0/791943824 conn(0x7f0fe80b3f40 msgr2=0x7f0fe80b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 shutdown_connections 2024-04-15T15:16:23.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.411+0000 7f0fef8eb640 1 -- 172.21.15.46:0/791943824 wait complete. 2024-04-15T15:16:23.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:23.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:23.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.531+0000 7fcaf7f82640 1 Processor -- start 2024-04-15T15:16:23.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.531+0000 7fcaf7f82640 1 -- start start 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf7f82640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcaf0067720 0x7fcaf006ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf7f82640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf015d660 con 0x7fcaf015d090 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf7f82640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf015d940 con 0x7fcaf0160b80 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf7f82640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf006f140 con 0x7fcaf0067720 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf551d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcaf0067720 0x7fcaf006ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf5d1e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcaf015d090 0x7fcaf006e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34904/0 (socket says 172.21.15.46:34904) 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf5d1e640 1 -- 172.21.15.46:0/2548455456 learned_addr learned my addr 172.21.15.46:0/2548455456 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf551d640 1 -- 172.21.15.46:0/2548455456 >> v1:172.21.15.78:6789/0 conn(0x7fcaf0160b80 legacy=0x7fcaf00ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf551d640 1 -- 172.21.15.46:0/2548455456 >> v1:172.21.15.46:6789/0 conn(0x7fcaf015d090 legacy=0x7fcaf006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf551d640 1 -- 172.21.15.46:0/2548455456 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf0160380 con 0x7fcaf0067720 2024-04-15T15:16:23.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.532+0000 7fcaf4d1c640 1 -- 172.21.15.46:0/2548455456 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768739433 0 0) 0x7fcaf015d660 con 0x7fcaf015d090 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.533+0000 7fcaf551d640 1 --2- 172.21.15.46:0/2548455456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0067720 0x7fcaf006ec00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcae00097a0 tx=0x7fcae002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f0adc91e8323a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.533+0000 7fcaf4d1c640 1 -- 172.21.15.46:0/2548455456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae00315a0 con 0x7fcaf0067720 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.533+0000 7fcaf4d1c640 1 -- 172.21.15.46:0/2548455456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae0037070 con 0x7fcaf0067720 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.533+0000 7fcaf4d1c640 1 -- 172.21.15.46:0/2548455456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae0031960 con 0x7fcaf0067720 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2548455456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0067720 msgr2=0x7fcaf006ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2548455456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0067720 0x7fcaf006ec00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcae00097a0 tx=0x7fcae002b840 comp rx=0 tx=0).stop 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2548455456 shutdown_connections 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2548455456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0067720 0x7fcaf006ec00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2548455456 >> 172.21.15.46:0/2548455456 conn(0x7fcaf0073360 msgr2=0x7fcaf006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2548455456 shutdown_connections 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.534+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2548455456 wait complete. 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.535+0000 7fcaf7f82640 1 Processor -- start 2024-04-15T15:16:23.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.535+0000 7fcaf7f82640 1 -- start start 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf7f82640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcaf0067720 0x7fcaf01483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf7f82640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcaf015d090 0x7fcaf00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf7f82640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 0x7fcaf00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf7f82640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0160ff0 con 0x7fcaf0160b80 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf7f82640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0160130 con 0x7fcaf015d090 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf7f82640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf00b0460 con 0x7fcaf0067720 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 0x7fcaf00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 0x7fcaf00ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36972/0 (socket says 172.21.15.46:36972) 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 -- 172.21.15.46:0/2999902364 learned_addr learned my addr 172.21.15.46:0/2999902364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf5d1e640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcaf0067720 0x7fcaf01483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf551d640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcaf015d090 0x7fcaf00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 -- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcaf0067720 msgr2=0x7fcaf01483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcaf0067720 0x7fcaf01483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 -- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcaf015d090 msgr2=0x7fcaf00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcaf015d090 0x7fcaf00cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 -- 172.21.15.46:0/2999902364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf00ce770 con 0x7fcaf0160b80 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcaf651f640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 0x7fcaf00ce0f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcae800b990 tx=0x7fcae800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8019070 con 0x7fcaf0160b80 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.536+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae800cba0 con 0x7fcaf0160b80 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.537+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf00cea50 con 0x7fcaf0160b80 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.537+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8014440 con 0x7fcaf0160b80 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.537+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf00cf010 con 0x7fcaf0160b80 2024-04-15T15:16:23.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.538+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf00689e0 con 0x7fcaf0160b80 2024-04-15T15:16:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.538+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcae801c070 con 0x7fcaf0160b80 2024-04-15T15:16:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.539+0000 7fcae67fc640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcad406c920 0x7fcad406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.539+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcae808bc30 con 0x7fcaf0160b80 2024-04-15T15:16:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.539+0000 7fcaf5d1e640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcad406c920 0x7fcad406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.539+0000 7fcaf5d1e640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcad406c920 0x7fcad406ede0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcaec000f60 tx=0x7fcaec007d50 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.543+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcae8059580 con 0x7fcaf0160b80 2024-04-15T15:16:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.726+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcaf006c690 con 0x7fcaf0160b80 2024-04-15T15:16:23.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.726+0000 7fcae67fc640 1 -- 172.21.15.46:0/2999902364 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fcae805d880 con 0x7fcaf0160b80 2024-04-15T15:16:23.728 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcad406c920 msgr2=0x7fcad406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcad406c920 0x7fcad406ede0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fcaec000f60 tx=0x7fcaec007d50 comp rx=0 tx=0).stop 2024-04-15T15:16:23.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 msgr2=0x7fcaf00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 0x7fcaf00ce0f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcae800b990 tx=0x7fcae800be50 comp rx=0 tx=0).stop 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 shutdown_connections 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcad406c920 0x7fcad406ede0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcaf0160b80 0x7fcaf00ce0f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcaf015d090 0x7fcaf00cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 --2- 172.21.15.46:0/2999902364 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcaf0067720 0x7fcaf01483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 >> 172.21.15.46:0/2999902364 conn(0x7fcaf0073360 msgr2=0x7fcaf006e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 shutdown_connections 2024-04-15T15:16:23.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.728+0000 7fcaf7f82640 1 -- 172.21.15.46:0/2999902364 wait complete. 2024-04-15T15:16:23.742 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:23.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.848+0000 7f2d8ca42640 1 Processor -- start 2024-04-15T15:16:23.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.848+0000 7f2d8ca42640 1 -- start start 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d8ca42640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8815c8d0 0x7f2d880cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d8ca42640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d8809b0e0 0x7f2d880b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d8ca42640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d880d0350 con 0x7f2d8809b650 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d8ca42640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2d880b5b00 con 0x7f2d8809b0e0 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d8ca42640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2d880ce100 con 0x7f2d8815c8d0 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d86d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d8809b0e0 0x7f2d880b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d85d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d8809b650 0x7f2d8809a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34908/0 (socket says 172.21.15.46:34908) 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.849+0000 7f2d85d9b640 1 -- 172.21.15.46:0/3811465406 learned_addr learned my addr 172.21.15.46:0/3811465406 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:23.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.850+0000 7f2d8659c640 1 --2- 172.21.15.46:0/3811465406 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8815c8d0 0x7f2d880cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.850+0000 7f2d86d9d640 1 -- 172.21.15.46:0/3811465406 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8815c8d0 msgr2=0x7f2d880cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.850+0000 7f2d86d9d640 1 --2- 172.21.15.46:0/3811465406 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8815c8d0 0x7f2d880cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.850+0000 7f2d86d9d640 1 -- 172.21.15.46:0/3811465406 >> v1:172.21.15.46:6789/0 conn(0x7f2d8809b650 legacy=0x7f2d8809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.850+0000 7f2d86d9d640 1 -- 172.21.15.46:0/3811465406 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d880b54b0 con 0x7f2d8809b0e0 2024-04-15T15:16:23.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.850+0000 7f2d8559a640 1 -- 172.21.15.46:0/3811465406 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3971356357 0 0) 0x7f2d880d0350 con 0x7f2d8809b650 2024-04-15T15:16:23.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d86d9d640 1 --2- 172.21.15.46:0/3811465406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880b5f90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009d90 tx=0x7f2d7802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b482361cd50c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8559a640 1 -- 172.21.15.46:0/3811465406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78039070 con 0x7f2d8809b0e0 2024-04-15T15:16:23.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8559a640 1 -- 172.21.15.46:0/3811465406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d7802cbc0 con 0x7f2d8809b0e0 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8559a640 1 -- 172.21.15.46:0/3811465406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78034440 con 0x7f2d8809b0e0 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 -- 172.21.15.46:0/3811465406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 msgr2=0x7f2d880b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/3811465406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880b5f90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009d90 tx=0x7f2d7802be10 comp rx=0 tx=0).stop 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 -- 172.21.15.46:0/3811465406 shutdown_connections 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/3811465406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880b5f90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/3811465406 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8815c8d0 0x7f2d880cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 -- 172.21.15.46:0/3811465406 >> 172.21.15.46:0/3811465406 conn(0x7f2d880b3fb0 msgr2=0x7f2d880b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 -- 172.21.15.46:0/3811465406 shutdown_connections 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.851+0000 7f2d8ca42640 1 -- 172.21.15.46:0/3811465406 wait complete. 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 Processor -- start 2024-04-15T15:16:23.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 -- start start 2024-04-15T15:16:23.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d8809b650 0x7f2d88151bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8809efc0 0x7f2d88154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d880b6650 con 0x7f2d8809b0e0 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2d880b64d0 con 0x7f2d8809b650 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.853+0000 7f2d8ca42640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2d880b67d0 con 0x7f2d8809efc0 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880c6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37004/0 (socket says 172.21.15.46:37004) 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 -- 172.21.15.46:0/195443684 learned_addr learned my addr 172.21.15.46:0/195443684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d86d9d640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8809efc0 0x7f2d88154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d85d9b640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d8809b650 0x7f2d88151bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 -- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8809efc0 msgr2=0x7f2d88154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8809efc0 0x7f2d88154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 -- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d8809b650 msgr2=0x7f2d88151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d8809b650 0x7f2d88151bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 -- 172.21.15.46:0/195443684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8809eb80 con 0x7f2d8809b0e0 2024-04-15T15:16:23.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d85d9b640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d8809b650 0x7f2d88151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d86d9d640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8809efc0 0x7f2d88154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8659c640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880c6ce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c7f0 tx=0x7f2d8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d80015070 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d80004220 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d880c73e0 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d80007620 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.854+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d880c79a0 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.855+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d880b3da0 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.856+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d800079d0 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.856+0000 7f2d7effd640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d6806c8f0 0x7f2d6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.856+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2d8008c270 con 0x7f2d8809b0e0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.856+0000 7f2d85d9b640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d6806c8f0 0x7f2d6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.857+0000 7f2d85d9b640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d6806c8f0 0x7f2d6806edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d880b31c0 tx=0x7f2d70007c10 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:23.861+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d880b3da0 con 0x7f2d8809b0e0 2024-04-15T15:16:24.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.045+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2d880690f0 con 0x7f2d8809b0e0 2024-04-15T15:16:24.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.045+0000 7f2d7effd640 1 -- 172.21.15.46:0/195443684 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f2d8005a2a0 con 0x7f2d8809b0e0 2024-04-15T15:16:24.047 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d6806c8f0 msgr2=0x7f2d6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d6806c8f0 0x7f2d6806edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d880b31c0 tx=0x7f2d70007c10 comp rx=0 tx=0).stop 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 msgr2=0x7f2d880c6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880c6ce0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c7f0 tx=0x7f2d8000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 shutdown_connections 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d6806c8f0 0x7f2d6806edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d8809efc0 0x7f2d88154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d8809b650 0x7f2d88151bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 --2- 172.21.15.46:0/195443684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d8809b0e0 0x7f2d880c6ce0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 >> 172.21.15.46:0/195443684 conn(0x7f2d880b3fb0 msgr2=0x7f2d880d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 shutdown_connections 2024-04-15T15:16:24.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.047+0000 7f2d8ca42640 1 -- 172.21.15.46:0/195443684 wait complete. 2024-04-15T15:16:24.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.166+0000 7f29535f2640 1 Processor -- start 2024-04-15T15:16:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.166+0000 7f29535f2640 1 -- start start 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.167+0000 7f29535f2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f294c0d3270 0x7f294c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.167+0000 7f29535f2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c0b47a0 con 0x7f294c15e700 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.167+0000 7f29535f2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f294c0b4a80 con 0x7f294c0d2360 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.167+0000 7f29535f2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f294c0b53b0 con 0x7f294c0d3270 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.167+0000 7f295138e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f294c15e700 0x7f294c0b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34916/0 (socket says 172.21.15.46:34916) 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.167+0000 7f295138e640 1 -- 172.21.15.46:0/2550941141 learned_addr learned my addr 172.21.15.46:0/2550941141 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f2951b8f640 1 --2- 172.21.15.46:0/2550941141 >> v2:172.21.15.78:3300/0 conn(0x7f294c0d3270 0x7f294c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108011587 0 0) 0x7f294c0b47a0 con 0x7f294c15e700 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2934003520 con 0x7f294c15e700 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762940834 0 0) 0x7f294c0b4a80 con 0x7f294c0d2360 2024-04-15T15:16:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c0b47a0 con 0x7f294c0d2360 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 103857175 0 0) 0x7f2934003520 con 0x7f294c15e700 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 >> v2:172.21.15.78:3300/0 conn(0x7f294c0d3270 msgr2=0x7f294c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 --2- 172.21.15.46:0/2550941141 >> v2:172.21.15.78:3300/0 conn(0x7f294c0d3270 0x7f294c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 >> v1:172.21.15.78:6789/0 conn(0x7f294c0d2360 legacy=0x7f294c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c0b4d60 con 0x7f294c15e700 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f29440017f0 con 0x7f294c15e700 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 >> v1:172.21.15.46:6789/0 conn(0x7f294c15e700 legacy=0x7f294c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2934003b40 0x7f2934003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2934004800 0x7f2934004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29340051c0 0x7f293400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f294c0b53b0 con 0x7f2934003b40 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f293400c060 con 0x7f29340051c0 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f293400c260 con 0x7f2934004800 2024-04-15T15:16:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f295138e640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2934003b40 0x7f2934003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f2950b8d640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2934004800 0x7f2934004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f2951b8f640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29340051c0 0x7f293400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f295138e640 1 -- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2934004800 msgr2=0x7f2934004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f295138e640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2934004800 0x7f2934004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f295138e640 1 -- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29340051c0 msgr2=0x7f293400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f295138e640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29340051c0 0x7f293400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f295138e640 1 -- 172.21.15.46:0/2550941141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2934003520 con 0x7f2934003b40 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.168+0000 7f2951b8f640 1 -- 172.21.15.46:0/2550941141 reap_dead start 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.169+0000 7f295138e640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2934003b40 0x7f2934003f60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f294400c160 tx=0x7f2934005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a910e89005e7a43c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.169+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944007ca0 con 0x7f2934003b40 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.169+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2944030070 con 0x7f2934003b40 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.169+0000 7f294bfff640 1 -- 172.21.15.46:0/2550941141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944007510 con 0x7f2934003b40 2024-04-15T15:16:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 -- 172.21.15.46:0/2550941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2934003b40 msgr2=0x7f2934003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2934003b40 0x7f2934003f60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f294400c160 tx=0x7f2934005830 comp rx=0 tx=0).stop 2024-04-15T15:16:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 -- 172.21.15.46:0/2550941141 shutdown_connections 2024-04-15T15:16:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 --2- 172.21.15.46:0/2550941141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2934003b40 0x7f2934003f60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 -- 172.21.15.46:0/2550941141 >> 172.21.15.46:0/2550941141 conn(0x7f294c08bf60 msgr2=0x7f294c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 -- 172.21.15.46:0/2550941141 shutdown_connections 2024-04-15T15:16:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.170+0000 7f29535f2640 1 -- 172.21.15.46:0/2550941141 wait complete. 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.171+0000 7f29535f2640 1 Processor -- start 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.171+0000 7f29535f2640 1 -- start start 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f29535f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2934003b40 0x7f294c153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f29535f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 0x7f294c1f7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f29535f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f294c1f8a70 0x7f294c1f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f29535f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f294c0d1d40 con 0x7f294c0b7600 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f29535f2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f294c0b4450 con 0x7f2934003b40 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f29535f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f294c0d1ec0 con 0x7f294c1f8a70 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 0x7f294c1f7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 0x7f294c1f7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37018/0 (socket says 172.21.15.46:37018) 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 -- 172.21.15.46:0/1918379632 learned_addr learned my addr 172.21.15.46:0/1918379632 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f295138e640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2934003b40 0x7f294c153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2951b8f640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f294c1f8a70 0x7f294c1f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 -- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f294c1f8a70 msgr2=0x7f294c1f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f294c1f8a70 0x7f294c1f8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 -- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2934003b40 msgr2=0x7f294c153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2934003b40 0x7f294c153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2950b8d640 1 -- 172.21.15.46:0/1918379632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c0ce780 con 0x7f294c0b7600 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.172+0000 7f2951b8f640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f294c1f8a70 0x7f294c1f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.173+0000 7f2950b8d640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 0x7f294c1f7e40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f294000e500 tx=0x7f294000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.173+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29400052e0 con 0x7f294c0b7600 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.173+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2940018070 con 0x7f294c0b7600 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.173+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940003830 con 0x7f294c0b7600 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.173+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c0ce1c0 con 0x7f294c0b7600 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.173+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c0cd850 con 0x7f294c0b7600 2024-04-15T15:16:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.174+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f294c15e060 con 0x7f294c0b7600 2024-04-15T15:16:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.174+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29400039d0 con 0x7f294c0b7600 2024-04-15T15:16:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.175+0000 7f2949ffb640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f293806c8f0 0x7f293806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.175+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f294008bd40 con 0x7f294c0b7600 2024-04-15T15:16:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.175+0000 7f295138e640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f293806c8f0 0x7f293806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.175+0000 7f295138e640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f293806c8f0 0x7f293806edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2944034000 tx=0x7f2944007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.179+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f294c15e060 con 0x7f294c0b7600 2024-04-15T15:16:24.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.363+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f294c0689e0 con 0x7f294c0b7600 2024-04-15T15:16:24.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.364+0000 7f2949ffb640 1 -- 172.21.15.46:0/1918379632 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f2940059d80 con 0x7f294c0b7600 2024-04-15T15:16:24.366 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f293806c8f0 msgr2=0x7f293806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f293806c8f0 0x7f293806edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2944034000 tx=0x7f2944007000 comp rx=0 tx=0).stop 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 msgr2=0x7f294c1f7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 0x7f294c1f7e40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f294000e500 tx=0x7f294000e9c0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 shutdown_connections 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f293806c8f0 0x7f293806edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f294c1f8a70 0x7f294c1f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f294c0b7600 0x7f294c1f7e40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 --2- 172.21.15.46:0/1918379632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2934003b40 0x7f294c153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 >> 172.21.15.46:0/1918379632 conn(0x7f294c08bf60 msgr2=0x7f294c0b5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 shutdown_connections 2024-04-15T15:16:24.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.366+0000 7f29535f2640 1 -- 172.21.15.46:0/1918379632 wait complete. 2024-04-15T15:16:24.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:24.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.484+0000 7f24b6c0c640 1 Processor -- start 2024-04-15T15:16:24.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.484+0000 7f24b6c0c640 1 -- start start 2024-04-15T15:16:24.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24b6c0c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24b00b5230 0x7f24b0072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24b6c0c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b0073080 con 0x7f24b0072c60 2024-04-15T15:16:24.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24b6c0c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b00b4190 con 0x7f24b0158770 2024-04-15T15:16:24.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24b6c0c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f24b00b4da0 con 0x7f24b00b5230 2024-04-15T15:16:24.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24affff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f24b00b5230 0x7f24b0072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24b51a9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f24b0072c60 0x7f24b006dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34924/0 (socket says 172.21.15.46:34924) 2024-04-15T15:16:24.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.485+0000 7f24b51a9640 1 -- 172.21.15.46:0/1327404759 learned_addr learned my addr 172.21.15.46:0/1327404759 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:24.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24affff640 1 -- 172.21.15.46:0/1327404759 >> v1:172.21.15.78:6789/0 conn(0x7f24b0158770 legacy=0x7f24b015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24affff640 1 -- 172.21.15.46:0/1327404759 >> v1:172.21.15.46:6789/0 conn(0x7f24b0072c60 legacy=0x7f24b006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24affff640 1 -- 172.21.15.46:0/1327404759 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b00b4470 con 0x7f24b00b5230 2024-04-15T15:16:24.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24af7fe640 1 -- 172.21.15.46:0/1327404759 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 376590340 0 0) 0x7f24b0073080 con 0x7f24b0072c60 2024-04-15T15:16:24.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24affff640 1 --2- 172.21.15.46:0/1327404759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b00b5230 0x7f24b0072720 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f24a40097a0 tx=0x7f24a402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9915688734aff63d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24af7fe640 1 -- 172.21.15.46:0/1327404759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4034070 con 0x7f24b00b5230 2024-04-15T15:16:24.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24af7fe640 1 -- 172.21.15.46:0/1327404759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a4030610 con 0x7f24b00b5230 2024-04-15T15:16:24.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.486+0000 7f24af7fe640 1 -- 172.21.15.46:0/1327404759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a40309e0 con 0x7f24b00b5230 2024-04-15T15:16:24.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 -- 172.21.15.46:0/1327404759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b00b5230 msgr2=0x7f24b0072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/1327404759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b00b5230 0x7f24b0072720 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f24a40097a0 tx=0x7f24a402b960 comp rx=0 tx=0).stop 2024-04-15T15:16:24.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 -- 172.21.15.46:0/1327404759 shutdown_connections 2024-04-15T15:16:24.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/1327404759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b00b5230 0x7f24b0072720 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 -- 172.21.15.46:0/1327404759 >> 172.21.15.46:0/1327404759 conn(0x7f24b006be00 msgr2=0x7f24b006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 -- 172.21.15.46:0/1327404759 shutdown_connections 2024-04-15T15:16:24.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.487+0000 7f24b6c0c640 1 -- 172.21.15.46:0/1327404759 wait complete. 2024-04-15T15:16:24.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.488+0000 7f24b6c0c640 1 Processor -- start 2024-04-15T15:16:24.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.488+0000 7f24b6c0c640 1 -- start start 2024-04-15T15:16:24.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b6c0c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f24b0072c60 0x7f24b0155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b6c0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24b00b5230 0x7f24b0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b6c0c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 0x7f24b014e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b6c0c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24b0071fa0 con 0x7f24b0158770 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b6c0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f24b00b4a30 con 0x7f24b00b5230 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b6c0c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f24b0158b50 con 0x7f24b0072c60 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b51a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 0x7f24b014e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b51a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 0x7f24b014e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37032/0 (socket says 172.21.15.46:37032) 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b51a9640 1 -- 172.21.15.46:0/206598210 learned_addr learned my addr 172.21.15.46:0/206598210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24affff640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24b00b5230 0x7f24b0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.489+0000 7f24b49a8640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f24b0072c60 0x7f24b0155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b51a9640 1 -- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f24b0072c60 msgr2=0x7f24b0155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b51a9640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f24b0072c60 0x7f24b0155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b51a9640 1 -- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24b00b5230 msgr2=0x7f24b0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b51a9640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24b00b5230 0x7f24b0155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b51a9640 1 -- 172.21.15.46:0/206598210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b014f4f0 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b49a8640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f24b0072c60 0x7f24b0155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b51a9640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 0x7f24b014e910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c700 tx=0x7f24a800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8018070 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a800ce10 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b014f7d0 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a800a880 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.490+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b015e160 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.491+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b00689e0 con 0x7f24b0158770 2024-04-15T15:16:24.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.492+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24a801b070 con 0x7f24b0158770 2024-04-15T15:16:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.492+0000 7f24ad7fa640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f249c06c8f0 0x7f249c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.492+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f24a808b410 con 0x7f24b0158770 2024-04-15T15:16:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.492+0000 7f24b49a8640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f249c06c8f0 0x7f249c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.493+0000 7f24b49a8640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f249c06c8f0 0x7f249c06edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24a0007fd0 tx=0x7f24a0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.496+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24a8058df0 con 0x7f24b0158770 2024-04-15T15:16:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.682+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f24b0159950 con 0x7f24b0158770 2024-04-15T15:16:24.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.682+0000 7f24ad7fa640 1 -- 172.21.15.46:0/206598210 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f24a805d0f0 con 0x7f24b0158770 2024-04-15T15:16:24.684 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f249c06c8f0 msgr2=0x7f249c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f249c06c8f0 0x7f249c06edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24a0007fd0 tx=0x7f24a0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 msgr2=0x7f24b014e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 0x7f24b014e910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c700 tx=0x7f24a800cbc0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 shutdown_connections 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f249c06c8f0 0x7f249c06edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24b0158770 0x7f24b014e910 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24b00b5230 0x7f24b0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 --2- 172.21.15.46:0/206598210 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f24b0072c60 0x7f24b0155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 >> 172.21.15.46:0/206598210 conn(0x7f24b006be00 msgr2=0x7f24b01579a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 shutdown_connections 2024-04-15T15:16:24.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.685+0000 7f24b6c0c640 1 -- 172.21.15.46:0/206598210 wait complete. 2024-04-15T15:16:24.698 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:24.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.801+0000 7f0d19e88640 1 Processor -- start 2024-04-15T15:16:24.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.801+0000 7f0d19e88640 1 -- start start 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d19e88640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d1409b0e0 0x7f0d14070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d19e88640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d1406ff30 con 0x7f0d140ce8e0 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d19e88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d140cecc0 con 0x7f0d14071d90 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d19e88640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0d14070820 con 0x7f0d1409b0e0 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d12ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d1409b0e0 0x7f0d14070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d137fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0d140ce8e0 0x7f0d1409a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34938/0 (socket says 172.21.15.46:34938) 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d12ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0d1409b0e0 0x7f0d14070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37044/0 (socket says 172.21.15.46:37044) 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1370342078 learned_addr learned my addr 172.21.15.46:0/1370342078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1370342078 >> v1:172.21.15.78:6789/0 conn(0x7f0d14071d90 legacy=0x7f0d140cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1370342078 >> v1:172.21.15.46:6789/0 conn(0x7f0d140ce8e0 legacy=0x7f0d1409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.802+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1370342078 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1409b9d0 con 0x7f0d1409b0e0 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.803+0000 7f0d12ffd640 1 --2- 172.21.15.46:0/1370342078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d14070cb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009870 tx=0x7f0d0802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb18622c0f4bb1d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.803+0000 7f0d127fc640 1 -- 172.21.15.46:0/1370342078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08035020 con 0x7f0d1409b0e0 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.803+0000 7f0d127fc640 1 -- 172.21.15.46:0/1370342078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d0802d040 con 0x7f0d1409b0e0 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.803+0000 7f0d127fc640 1 -- 172.21.15.46:0/1370342078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08031a80 con 0x7f0d1409b0e0 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 -- 172.21.15.46:0/1370342078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 msgr2=0x7f0d14070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1370342078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d14070cb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009870 tx=0x7f0d0802b900 comp rx=0 tx=0).stop 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 -- 172.21.15.46:0/1370342078 shutdown_connections 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1370342078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d14070cb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 -- 172.21.15.46:0/1370342078 >> 172.21.15.46:0/1370342078 conn(0x7f0d140b3f40 msgr2=0x7f0d140b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 -- 172.21.15.46:0/1370342078 shutdown_connections 2024-04-15T15:16:24.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.804+0000 7f0d19e88640 1 -- 172.21.15.46:0/1370342078 wait complete. 2024-04-15T15:16:24.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.805+0000 7f0d19e88640 1 Processor -- start 2024-04-15T15:16:24.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.805+0000 7f0d19e88640 1 -- start start 2024-04-15T15:16:24.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d19e88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d14071d90 0x7f0d140b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d19e88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d141e7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d19e88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0d140ce8e0 0x7f0d1415dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d19e88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d140704b0 con 0x7f0d1409b0e0 2024-04-15T15:16:24.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d19e88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0d140701d0 con 0x7f0d14071d90 2024-04-15T15:16:24.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d19e88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0d14071970 con 0x7f0d140ce8e0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d141e7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d141e7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37058/0 (socket says 172.21.15.46:37058) 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1259659670 learned_addr learned my addr 172.21.15.46:0/1259659670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d137fe640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d14071d90 0x7f0d140b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d13fff640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0d140ce8e0 0x7f0d1415dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0d140ce8e0 msgr2=0x7f0d1415dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0d140ce8e0 0x7f0d1415dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d14071d90 msgr2=0x7f0d140b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d14071d90 0x7f0d140b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 -- 172.21.15.46:0/1259659670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d140b6aa0 con 0x7f0d1409b0e0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d137fe640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d14071d90 0x7f0d140b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.806+0000 7f0d12ffd640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d141e7cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009870 tx=0x7f0d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.807+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0803ac60 con 0x7f0d1409b0e0 2024-04-15T15:16:24.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.807+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d0803ae00 con 0x7f0d1409b0e0 2024-04-15T15:16:24.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.807+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d140b6d80 con 0x7f0d1409b0e0 2024-04-15T15:16:24.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.807+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08044630 con 0x7f0d1409b0e0 2024-04-15T15:16:24.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.807+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d140b7340 con 0x7f0d1409b0e0 2024-04-15T15:16:24.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.808+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d140689e0 con 0x7f0d1409b0e0 2024-04-15T15:16:24.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.808+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d0804b060 con 0x7f0d1409b0e0 2024-04-15T15:16:24.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.808+0000 7f0cfbfff640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0d0006c8f0 0x7f0d0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.809+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0d080adfa0 con 0x7f0d1409b0e0 2024-04-15T15:16:24.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.809+0000 7f0d137fe640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0d0006c8f0 0x7f0d0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.809+0000 7f0d137fe640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0d0006c8f0 0x7f0d0006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0d04000f60 tx=0x7f0d04007ce0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.813+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d0803f070 con 0x7f0d1409b0e0 2024-04-15T15:16:24.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.996+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d1409aec0 con 0x7f0d1409b0e0 2024-04-15T15:16:24.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.997+0000 7f0cfbfff640 1 -- 172.21.15.46:0/1259659670 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f0d0807c060 con 0x7f0d1409b0e0 2024-04-15T15:16:24.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:24.999 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:25.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0d0006c8f0 msgr2=0x7f0d0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0d0006c8f0 0x7f0d0006edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0d04000f60 tx=0x7f0d04007ce0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 msgr2=0x7f0d141e7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d141e7cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009870 tx=0x7f0d080046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 shutdown_connections 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0d0006c8f0 0x7f0d0006edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0d140ce8e0 0x7f0d1415dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d1409b0e0 0x7f0d141e7cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 --2- 172.21.15.46:0/1259659670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d14071d90 0x7f0d140b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:24.999+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 >> 172.21.15.46:0/1259659670 conn(0x7f0d140b3f40 msgr2=0x7f0d140af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.000+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 shutdown_connections 2024-04-15T15:16:25.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.000+0000 7f0d19e88640 1 -- 172.21.15.46:0/1259659670 wait complete. 2024-04-15T15:16:25.013 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:25.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:25.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.118+0000 7fc82a700640 1 Processor -- start 2024-04-15T15:16:25.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.118+0000 7fc82a700640 1 -- start start 2024-04-15T15:16:25.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc82a700640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc82406c0f0 0x7fc8240ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc82a700640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8240cbd10 con 0x7fc82415eb00 2024-04-15T15:16:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc82a700640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8240cbff0 con 0x7fc8240b0560 2024-04-15T15:16:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc82a700640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc82415eee0 con 0x7fc82406c0f0 2024-04-15T15:16:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc828c9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc82406c0f0 0x7fc8240ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc823fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc82415eb00 0x7fc8240ae2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34954/0 (socket says 172.21.15.46:34954) 2024-04-15T15:16:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.119+0000 7fc823fff640 1 -- 172.21.15.46:0/470518255 learned_addr learned my addr 172.21.15.46:0/470518255 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:25.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.120+0000 7fc828c9d640 1 -- 172.21.15.46:0/470518255 >> v1:172.21.15.78:6789/0 conn(0x7fc8240b0560 legacy=0x7fc82406b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.120+0000 7fc828c9d640 1 -- 172.21.15.46:0/470518255 >> v1:172.21.15.46:6789/0 conn(0x7fc82415eb00 legacy=0x7fc8240ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.120+0000 7fc828c9d640 1 -- 172.21.15.46:0/470518255 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8240cc2d0 con 0x7fc82406c0f0 2024-04-15T15:16:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.120+0000 7fc828c9d640 1 --2- 172.21.15.46:0/470518255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82406c0f0 0x7fc8240ccdb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009d90 tx=0x7fc81c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561ead031767eacd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc822ffd640 1 -- 172.21.15.46:0/470518255 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c039070 con 0x7fc82406c0f0 2024-04-15T15:16:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc822ffd640 1 -- 172.21.15.46:0/470518255 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81c02cc60 con 0x7fc82406c0f0 2024-04-15T15:16:25.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc822ffd640 1 -- 172.21.15.46:0/470518255 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c034660 con 0x7fc82406c0f0 2024-04-15T15:16:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 -- 172.21.15.46:0/470518255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82406c0f0 msgr2=0x7fc8240ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 --2- 172.21.15.46:0/470518255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82406c0f0 0x7fc8240ccdb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009d90 tx=0x7fc81c02beb0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 -- 172.21.15.46:0/470518255 shutdown_connections 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 --2- 172.21.15.46:0/470518255 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82406c0f0 0x7fc8240ccdb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 -- 172.21.15.46:0/470518255 >> 172.21.15.46:0/470518255 conn(0x7fc82409a9d0 msgr2=0x7fc8240d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 -- 172.21.15.46:0/470518255 shutdown_connections 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.121+0000 7fc82a700640 1 -- 172.21.15.46:0/470518255 wait complete. 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.122+0000 7fc82a700640 1 Processor -- start 2024-04-15T15:16:25.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 -- start start 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82406c0f0 0x7fc8240cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc8240b0560 0x7fc8240b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 0x7fc8241f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240cec60 con 0x7fc82415eb00 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc8240cc890 con 0x7fc82406c0f0 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc82a700640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc8240cede0 con 0x7fc8240b0560 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc828c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 0x7fc8241f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc828c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 0x7fc8241f8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37074/0 (socket says 172.21.15.46:37074) 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.123+0000 7fc828c9d640 1 -- 172.21.15.46:0/813214590 learned_addr learned my addr 172.21.15.46:0/813214590 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc823fff640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82406c0f0 0x7fc8240cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc8237fe640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc8240b0560 0x7fc8240b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc828c9d640 1 -- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc8240b0560 msgr2=0x7fc8240b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc828c9d640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc8240b0560 0x7fc8240b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc828c9d640 1 -- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82406c0f0 msgr2=0x7fc8240cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc828c9d640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82406c0f0 0x7fc8240cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc828c9d640 1 -- 172.21.15.46:0/813214590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8240b5c30 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc823fff640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82406c0f0 0x7fc8240cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc828c9d640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 0x7fc8241f8100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc81c037040 tx=0x7fc81c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c044030 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc81c02ca60 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c039040 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8240b5f10 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.124+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8240c8e60 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.125+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8240d09a0 con 0x7fc82415eb00 2024-04-15T15:16:25.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.126+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc81c047070 con 0x7fc82415eb00 2024-04-15T15:16:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.126+0000 7fc820ff9640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc81006c8f0 0x7fc81006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.126+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc81c0ae7b0 con 0x7fc82415eb00 2024-04-15T15:16:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.126+0000 7fc823fff640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc81006c8f0 0x7fc81006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.127+0000 7fc823fff640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc81006c8f0 0x7fc81006edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc814000f60 tx=0x7fc814009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.130+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc81c04e070 con 0x7fc82415eb00 2024-04-15T15:16:25.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.314+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8240690f0 con 0x7fc82415eb00 2024-04-15T15:16:25.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.315+0000 7fc820ff9640 1 -- 172.21.15.46:0/813214590 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fc81c07c930 con 0x7fc82415eb00 2024-04-15T15:16:25.317 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc81006c8f0 msgr2=0x7fc81006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc81006c8f0 0x7fc81006edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc814000f60 tx=0x7fc814009040 comp rx=0 tx=0).stop 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 msgr2=0x7fc8241f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 0x7fc8241f8100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc81c037040 tx=0x7fc81c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 shutdown_connections 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc81006c8f0 0x7fc81006edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc82415eb00 0x7fc8241f8100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc8240b0560 0x7fc8240b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 --2- 172.21.15.46:0/813214590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc82406c0f0 0x7fc8240cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 >> 172.21.15.46:0/813214590 conn(0x7fc82409a9d0 msgr2=0x7fc8240ae2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 shutdown_connections 2024-04-15T15:16:25.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.317+0000 7fc82a700640 1 -- 172.21.15.46:0/813214590 wait complete. 2024-04-15T15:16:25.330 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:25.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.435+0000 7f8ca1bf4640 1 Processor -- start 2024-04-15T15:16:25.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.435+0000 7f8ca1bf4640 1 -- start start 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.436+0000 7f8ca1bf4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 0x7f8c9c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.436+0000 7f8ca1bf4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c9c0d1cf0 0x7f8c9c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.436+0000 7f8ca1bf4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c9c15eb70 con 0x7f8c9c0b7be0 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.436+0000 7f8ca1bf4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f8c9c0d3790 con 0x7f8c9c0d1cf0 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.436+0000 7f8ca1bf4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8c9c0d45b0 con 0x7f8c9c15e790 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 0x7f8c9c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c9c0d1cf0 0x7f8c9c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c9c0d1cf0 0x7f8c9c0d4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:40896/0 (socket says 172.21.15.46:40896) 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8c9c0b7be0 0x7f8c9c0d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:55966/0 (socket says 172.21.15.46:55966) 2024-04-15T15:16:25.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 -- 172.21.15.46:0/3749988384 learned_addr learned my addr 172.21.15.46:0/3749988384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9b7fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 0x7f8c9c0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:36818/0 (socket says 172.21.15.46:36818) 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 -- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 msgr2=0x7f8c9c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 --2- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 0x7f8c9c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 -- 172.21.15.46:0/3749988384 >> v1:172.21.15.78:6789/0 conn(0x7f8c9c0b7be0 legacy=0x7f8c9c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9bfff640 1 -- 172.21.15.46:0/3749988384 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c0d3c80 con 0x7f8c9c0d1cf0 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9a7fc640 1 -- 172.21.15.46:0/3749988384 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618140514 0 0) 0x7f8c9c15eb70 con 0x7f8c9c0b7be0 2024-04-15T15:16:25.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.437+0000 7f8c9b7fe640 1 --2- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 0x7f8c9c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8c9bfff640 1 --2- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c0d4a40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0d3910 tx=0x7f8c9402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ed1c496c06918c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8c9a7fc640 1 -- 172.21.15.46:0/3749988384 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9402c9c0 con 0x7f8c9c0d1cf0 2024-04-15T15:16:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8c9a7fc640 1 -- 172.21.15.46:0/3749988384 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9402cb60 con 0x7f8c9c0d1cf0 2024-04-15T15:16:25.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8c9a7fc640 1 -- 172.21.15.46:0/3749988384 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c940343f0 con 0x7f8c9c0d1cf0 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 msgr2=0x7f8c9c0d4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c0d4a40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0d3910 tx=0x7f8c9402be20 comp rx=0 tx=0).stop 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/3749988384 shutdown_connections 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c0d4a40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/3749988384 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c15e790 0x7f8c9c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.438+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/3749988384 >> 172.21.15.46:0/3749988384 conn(0x7f8c9c067b40 msgr2=0x7f8c9c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.439+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/3749988384 shutdown_connections 2024-04-15T15:16:25.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.439+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/3749988384 wait complete. 2024-04-15T15:16:25.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.440+0000 7f8ca1bf4640 1 Processor -- start 2024-04-15T15:16:25.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.440+0000 7f8ca1bf4640 1 -- start start 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.440+0000 7f8ca1bf4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c0b7be0 0x7f8c9c1612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.440+0000 7f8ca1bf4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c1f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8ca1bf4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 0x7f8c9c159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8ca1bf4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c0d4f80 con 0x7f8c9c15e790 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8ca1bf4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9c0d4240 con 0x7f8c9c0d1cf0 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8ca1bf4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8c9c0d5100 con 0x7f8c9c0b7be0 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 0x7f8c9c159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 0x7f8c9c159d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37088/0 (socket says 172.21.15.46:37088) 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 -- 172.21.15.46:0/1783946299 learned_addr learned my addr 172.21.15.46:0/1783946299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9affd640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c1f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9b7fe640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c0b7be0 0x7f8c9c1612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 -- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c0b7be0 msgr2=0x7f8c9c1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c0b7be0 0x7f8c9c1612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 -- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 msgr2=0x7f8c9c1f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c1f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 -- 172.21.15.46:0/1783946299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9c160830 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c9bfff640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 0x7f8c9c159d10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c9402c430 tx=0x7f8c94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94043020 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9403e450 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9c1596d0 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9403e840 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.441+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c9c160dd0 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.443+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c9c0689e0 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.443+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c94046020 con 0x7f8c9c15e790 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.443+0000 7f8c83fff640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c8806c8f0 0x7f8c8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.443+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8c940ae840 con 0x7f8c9c15e790 2024-04-15T15:16:25.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.444+0000 7f8c9b7fe640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c8806c8f0 0x7f8c8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.444+0000 7f8c9b7fe640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c8806c8f0 0x7f8c8806edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0b74f0 tx=0x7f8c8c009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.448+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c9404d020 con 0x7f8c9c15e790 2024-04-15T15:16:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.634+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c9c15f860 con 0x7f8c9c15e790 2024-04-15T15:16:25.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.634+0000 7f8c83fff640 1 -- 172.21.15.46:0/1783946299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f8c9407c2b0 con 0x7f8c9c15e790 2024-04-15T15:16:25.636 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c8806c8f0 msgr2=0x7f8c8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c8806c8f0 0x7f8c8806edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0b74f0 tx=0x7f8c8c009190 comp rx=0 tx=0).stop 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 msgr2=0x7f8c9c159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 0x7f8c9c159d10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c9402c430 tx=0x7f8c94004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 shutdown_connections 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c8806c8f0 0x7f8c8806edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c9c15e790 0x7f8c9c159d10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c9c0d1cf0 0x7f8c9c1f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 --2- 172.21.15.46:0/1783946299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c9c0b7be0 0x7f8c9c1612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 >> 172.21.15.46:0/1783946299 conn(0x7f8c9c067b40 msgr2=0x7f8c9c0b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.636+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 shutdown_connections 2024-04-15T15:16:25.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.637+0000 7f8ca1bf4640 1 -- 172.21.15.46:0/1783946299 wait complete. 2024-04-15T15:16:25.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:25.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.758+0000 7fd32467b640 1 Processor -- start 2024-04-15T15:16:25.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.758+0000 7fd32467b640 1 -- start start 2024-04-15T15:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd32467b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd31c158770 0x7fd31c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd32467b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd31c072e30 0x7fd31c06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd32467b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd31c15c1d0 con 0x7fd31c0b5230 2024-04-15T15:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd32467b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c0b4da0 con 0x7fd31c158770 2024-04-15T15:16:25.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd32467b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c0b4190 con 0x7fd31c072e30 2024-04-15T15:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd31c158770 0x7fd31c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd31c158770 0x7fd31c15bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37090/0 (socket says 172.21.15.46:37090) 2024-04-15T15:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 -- 172.21.15.46:0/3212134788 learned_addr learned my addr 172.21.15.46:0/3212134788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd321c16640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd31c0b5230 0x7fd31c072720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34956/0 (socket says 172.21.15.46:34956) 2024-04-15T15:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322c18640 1 --2- 172.21.15.46:0/3212134788 >> v2:172.21.15.78:3300/0 conn(0x7fd31c072e30 0x7fd31c06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 -- 172.21.15.46:0/3212134788 >> v2:172.21.15.78:3300/0 conn(0x7fd31c072e30 msgr2=0x7fd31c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 --2- 172.21.15.46:0/3212134788 >> v2:172.21.15.78:3300/0 conn(0x7fd31c072e30 0x7fd31c06dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 -- 172.21.15.46:0/3212134788 >> v1:172.21.15.46:6789/0 conn(0x7fd31c0b5230 legacy=0x7fd31c072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322417640 1 -- 172.21.15.46:0/3212134788 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c0b4470 con 0x7fd31c158770 2024-04-15T15:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd321415640 1 -- 172.21.15.46:0/3212134788 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688043422 0 0) 0x7fd30c001110 con 0x7fd31c0b5230 2024-04-15T15:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.759+0000 7fd322c18640 1 --2- 172.21.15.46:0/3212134788 >> v2:172.21.15.78:3300/0 conn(0x7fd31c072e30 0x7fd31c06dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:25.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.760+0000 7fd322417640 1 --2- 172.21.15.46:0/3212134788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c158770 0x7fd31c15bc90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd31c072640 tx=0x7fd31802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a15f060c45a820b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.760+0000 7fd321415640 1 -- 172.21.15.46:0/3212134788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318034070 con 0x7fd31c158770 2024-04-15T15:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.760+0000 7fd321415640 1 -- 172.21.15.46:0/3212134788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd318030860 con 0x7fd31c158770 2024-04-15T15:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.760+0000 7fd321415640 1 -- 172.21.15.46:0/3212134788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd318030c30 con 0x7fd31c158770 2024-04-15T15:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 -- 172.21.15.46:0/3212134788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c158770 msgr2=0x7fd31c15bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 --2- 172.21.15.46:0/3212134788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c158770 0x7fd31c15bc90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd31c072640 tx=0x7fd31802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 -- 172.21.15.46:0/3212134788 shutdown_connections 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 --2- 172.21.15.46:0/3212134788 >> v2:172.21.15.78:3300/0 conn(0x7fd31c072e30 0x7fd31c06dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 --2- 172.21.15.46:0/3212134788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c158770 0x7fd31c15bc90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 -- 172.21.15.46:0/3212134788 >> 172.21.15.46:0/3212134788 conn(0x7fd31c06be00 msgr2=0x7fd31c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 -- 172.21.15.46:0/3212134788 shutdown_connections 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.761+0000 7fd32467b640 1 -- 172.21.15.46:0/3212134788 wait complete. 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.762+0000 7fd32467b640 1 Processor -- start 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.762+0000 7fd32467b640 1 -- start start 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd31c072e30 0x7fd31c1f5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 0x7fd31c099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd31c158770 0x7fd31c146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c071fa0 con 0x7fd31c0b5230 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c0b4a30 con 0x7fd31c158770 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c158b50 con 0x7fd31c072e30 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 0x7fd31c099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 0x7fd31c099890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37106/0 (socket says 172.21.15.46:37106) 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 -- 172.21.15.46:0/2429614675 learned_addr learned my addr 172.21.15.46:0/2429614675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd322c18640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd31c158770 0x7fd31c146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd322417640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd31c072e30 0x7fd31c1f5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 -- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd31c072e30 msgr2=0x7fd31c1f5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd31c072e30 0x7fd31c1f5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 -- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd31c158770 msgr2=0x7fd31c146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd31c158770 0x7fd31c146190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 -- 172.21.15.46:0/2429614675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c1f58c0 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd322c18640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd31c158770 0x7fd31c146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd321c16640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 0x7fd31c099890 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e750 tx=0x7fd30c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00ee60 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd30c016020 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c146be0 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c007520 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.763+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c1f4b80 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.764+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c1f49a0 con 0x7fd31c0b5230 2024-04-15T15:16:25.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.765+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd30c00c7f0 con 0x7fd31c0b5230 2024-04-15T15:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.765+0000 7fd312ffd640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd30006c8f0 0x7fd30006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.766+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd30c08e590 con 0x7fd31c0b5230 2024-04-15T15:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.766+0000 7fd322417640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd30006c8f0 0x7fd30006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.766+0000 7fd322417640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd30006c8f0 0x7fd30006edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd318036000 tx=0x7fd31803b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.770+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd31c1f49a0 con 0x7fd31c0b5230 2024-04-15T15:16:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.957+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd31c0689e0 con 0x7fd31c0b5230 2024-04-15T15:16:25.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.958+0000 7fd312ffd640 1 -- 172.21.15.46:0/2429614675 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fd30c020020 con 0x7fd31c0b5230 2024-04-15T15:16:25.959 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:25.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.959+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd30006c8f0 msgr2=0x7fd30006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.959+0000 7fd32467b640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd30006c8f0 0x7fd30006edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd318036000 tx=0x7fd31803b040 comp rx=0 tx=0).stop 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.959+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 msgr2=0x7fd31c099890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.959+0000 7fd32467b640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 0x7fd31c099890 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e750 tx=0x7fd30c00ec10 comp rx=0 tx=0).stop 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 shutdown_connections 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd30006c8f0 0x7fd30006edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd31c158770 0x7fd31c146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd31c0b5230 0x7fd31c099890 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 --2- 172.21.15.46:0/2429614675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd31c072e30 0x7fd31c1f5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 >> 172.21.15.46:0/2429614675 conn(0x7fd31c06be00 msgr2=0x7fd31c0b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 shutdown_connections 2024-04-15T15:16:25.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:25.960+0000 7fd32467b640 1 -- 172.21.15.46:0/2429614675 wait complete. 2024-04-15T15:16:25.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:26.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.079+0000 7f3a2659d640 1 Processor -- start 2024-04-15T15:16:26.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a2659d640 1 -- start start 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a2659d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3a200ae5e0 0x7f3a200afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a2659d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a200b0260 0x7f3a2015fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a2659d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a2015c8d0 con 0x7f3a200706f0 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a2659d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3a2015d4c0 con 0x7f3a200b0260 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a2659d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3a2015cb90 con 0x7f3a200ae5e0 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a25d9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a200b0260 0x7f3a2015fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a25d9c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3a200b0260 0x7f3a2015fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37118/0 (socket says 172.21.15.46:37118) 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.080+0000 7f3a25d9c640 1 -- 172.21.15.46:0/1359613735 learned_addr learned my addr 172.21.15.46:0/1359613735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.081+0000 7f3a24d9a640 1 --2- 172.21.15.46:0/1359613735 >> v2:172.21.15.78:3300/0 conn(0x7f3a200ae5e0 0x7f3a200afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.081+0000 7f3a25d9c640 1 -- 172.21.15.46:0/1359613735 >> v2:172.21.15.78:3300/0 conn(0x7f3a200ae5e0 msgr2=0x7f3a200afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.081+0000 7f3a25d9c640 1 --2- 172.21.15.46:0/1359613735 >> v2:172.21.15.78:3300/0 conn(0x7f3a200ae5e0 0x7f3a200afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.081+0000 7f3a25d9c640 1 -- 172.21.15.46:0/1359613735 >> v1:172.21.15.78:6789/0 conn(0x7f3a200706f0 legacy=0x7f3a200aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.081+0000 7f3a25d9c640 1 -- 172.21.15.46:0/1359613735 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2015ce70 con 0x7f3a200b0260 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a25d9c640 1 --2- 172.21.15.46:0/1359613735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2015fec0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3a200afa90 tx=0x7f3a1002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3297e4a9d9ebce56 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a1ffff640 1 -- 172.21.15.46:0/1359613735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1002cb20 con 0x7f3a200b0260 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a1ffff640 1 -- 172.21.15.46:0/1359613735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1002ccc0 con 0x7f3a200b0260 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a1ffff640 1 -- 172.21.15.46:0/1359613735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10034610 con 0x7f3a200b0260 2024-04-15T15:16:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 -- 172.21.15.46:0/1359613735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 msgr2=0x7f3a2015fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 --2- 172.21.15.46:0/1359613735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2015fec0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3a200afa90 tx=0x7f3a1002bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 -- 172.21.15.46:0/1359613735 shutdown_connections 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 --2- 172.21.15.46:0/1359613735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2015fec0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 --2- 172.21.15.46:0/1359613735 >> v2:172.21.15.78:3300/0 conn(0x7f3a200ae5e0 0x7f3a200afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 -- 172.21.15.46:0/1359613735 >> 172.21.15.46:0/1359613735 conn(0x7f3a2006ccd0 msgr2=0x7f3a2006d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 -- 172.21.15.46:0/1359613735 shutdown_connections 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.082+0000 7f3a2659d640 1 -- 172.21.15.46:0/1359613735 wait complete. 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.083+0000 7f3a2659d640 1 Processor -- start 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 -- start start 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3a200706f0 0x7f3a20150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a200ae5e0 0x7f3a20157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2009ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3a200b06b0 con 0x7f3a200b0260 2024-04-15T15:16:26.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2006eac0 con 0x7f3a200ae5e0 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2659d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3a200b0830 con 0x7f3a200706f0 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a25d9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2009ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a25d9c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2009ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37122/0 (socket says 172.21.15.46:37122) 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a25d9c640 1 -- 172.21.15.46:0/4177477222 learned_addr learned my addr 172.21.15.46:0/4177477222 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a24d9a640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a200ae5e0 0x7f3a20157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.084+0000 7f3a2559b640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3a200706f0 0x7f3a20150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a25d9c640 1 -- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3a200706f0 msgr2=0x7f3a20150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a25d9c640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3a200706f0 0x7f3a20150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a25d9c640 1 -- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a200ae5e0 msgr2=0x7f3a20157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a25d9c640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a200ae5e0 0x7f3a20157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a25d9c640 1 -- 172.21.15.46:0/4177477222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a201518b0 con 0x7f3a200b0260 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a25d9c640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2009ded0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3a10036040 tx=0x7f3a10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10043020 con 0x7f3a200b0260 2024-04-15T15:16:26.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1003e450 con 0x7f3a200b0260 2024-04-15T15:16:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a20151b90 con 0x7f3a200b0260 2024-04-15T15:16:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10009510 con 0x7f3a200b0260 2024-04-15T15:16:26.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.085+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a20152150 con 0x7f3a200b0260 2024-04-15T15:16:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.086+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a20153fa0 con 0x7f3a200b0260 2024-04-15T15:16:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.087+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a10046020 con 0x7f3a200b0260 2024-04-15T15:16:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.087+0000 7f3a1dffb640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3a0c06c8f0 0x7f3a0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.087+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3a100ae730 con 0x7f3a200b0260 2024-04-15T15:16:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.087+0000 7f3a2559b640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3a0c06c8f0 0x7f3a0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.088+0000 7f3a2559b640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3a0c06c8f0 0x7f3a0c06edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a18007fd0 tx=0x7f3a18009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.092+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a1004d030 con 0x7f3a200b0260 2024-04-15T15:16:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.278+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3a2004fd50 con 0x7f3a200b0260 2024-04-15T15:16:26.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.278+0000 7f3a1dffb640 1 -- 172.21.15.46:0/4177477222 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3a1007c760 con 0x7f3a200b0260 2024-04-15T15:16:26.280 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3a0c06c8f0 msgr2=0x7f3a0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3a0c06c8f0 0x7f3a0c06edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3a18007fd0 tx=0x7f3a18009040 comp rx=0 tx=0).stop 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 msgr2=0x7f3a2009ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2009ded0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3a10036040 tx=0x7f3a10004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 shutdown_connections 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3a0c06c8f0 0x7f3a0c06edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a200b0260 0x7f3a2009ded0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a200ae5e0 0x7f3a20157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 --2- 172.21.15.46:0/4177477222 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3a200706f0 0x7f3a20150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 >> 172.21.15.46:0/4177477222 conn(0x7f3a2006ccd0 msgr2=0x7f3a2006f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 shutdown_connections 2024-04-15T15:16:26.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.281+0000 7f3a2659d640 1 -- 172.21.15.46:0/4177477222 wait complete. 2024-04-15T15:16:26.306 INFO:tasks.cephfs_test_runner:Starting test: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:16:26.306 INFO:tasks.cephfs_test_runner:test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:16:26.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:26.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.410+0000 7fd8eede8640 1 Processor -- start 2024-04-15T15:16:26.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.410+0000 7fd8eede8640 1 -- start start 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8eede8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8e80d5150 0x7fd8e80d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8eede8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e815eb70 con 0x7fd8e815e790 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8eede8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e80d3c80 con 0x7fd8e80d15e0 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8eede8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd8e80d3790 con 0x7fd8e80d5150 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8ed385640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8e80d5150 0x7fd8e80d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8ecb84640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd8e815e790 0x7fd8e80b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34960/0 (socket says 172.21.15.46:34960) 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8ecb84640 1 -- 172.21.15.46:0/217758510 learned_addr learned my addr 172.21.15.46:0/217758510 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8ed385640 1 -- 172.21.15.46:0/217758510 >> v1:172.21.15.78:6789/0 conn(0x7fd8e80d15e0 legacy=0x7fd8e80d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8ed385640 1 -- 172.21.15.46:0/217758510 >> v1:172.21.15.46:6789/0 conn(0x7fd8e815e790 legacy=0x7fd8e80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8ed385640 1 -- 172.21.15.46:0/217758510 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e80d3f60 con 0x7fd8e80d5150 2024-04-15T15:16:26.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.411+0000 7fd8df7fe640 1 -- 172.21.15.46:0/217758510 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515633429 0 0) 0x7fd8e815eb70 con 0x7fd8e815e790 2024-04-15T15:16:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.412+0000 7fd8ed385640 1 --2- 172.21.15.46:0/217758510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d5150 0x7fd8e80d1a90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6338e8764d694221 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.412+0000 7fd8df7fe640 1 -- 172.21.15.46:0/217758510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00315a0 con 0x7fd8e80d5150 2024-04-15T15:16:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.412+0000 7fd8df7fe640 1 -- 172.21.15.46:0/217758510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e0037070 con 0x7fd8e80d5150 2024-04-15T15:16:26.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.412+0000 7fd8df7fe640 1 -- 172.21.15.46:0/217758510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031960 con 0x7fd8e80d5150 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 -- 172.21.15.46:0/217758510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d5150 msgr2=0x7fd8e80d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 --2- 172.21.15.46:0/217758510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d5150 0x7fd8e80d1a90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b840 comp rx=0 tx=0).stop 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 -- 172.21.15.46:0/217758510 shutdown_connections 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 --2- 172.21.15.46:0/217758510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d5150 0x7fd8e80d1a90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 -- 172.21.15.46:0/217758510 >> 172.21.15.46:0/217758510 conn(0x7fd8e8067b40 msgr2=0x7fd8e8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 -- 172.21.15.46:0/217758510 shutdown_connections 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.413+0000 7fd8eede8640 1 -- 172.21.15.46:0/217758510 wait complete. 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.414+0000 7fd8eede8640 1 Processor -- start 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.414+0000 7fd8eede8640 1 -- start start 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 0x7fd8e8154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8e80d5150 0x7fd8e80cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8e815e790 0x7fd8e80cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e80b7db0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e80d45b0 con 0x7fd8e815e790 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e80d29c0 con 0x7fd8e80d5150 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 0x7fd8e8154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 0x7fd8e8154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37140/0 (socket says 172.21.15.46:37140) 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 -- 172.21.15.46:0/3287337596 learned_addr learned my addr 172.21.15.46:0/3287337596 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ed385640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8e815e790 0x7fd8e80cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8dffff640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8e80d5150 0x7fd8e80cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 -- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8e80d5150 msgr2=0x7fd8e80cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8e80d5150 0x7fd8e80cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 -- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8e815e790 msgr2=0x7fd8e80cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8e815e790 0x7fd8e80cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 -- 172.21.15.46:0/3287337596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e80cddf0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ed385640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8e815e790 0x7fd8e80cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8dffff640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8e80d5150 0x7fd8e80cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8ecb84640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 0x7fd8e8154430 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c780 tx=0x7fd8e400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4018070 con 0x7fd8e80d15e0 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e400a640 con 0x7fd8e80d15e0 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e400aab0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.415+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e81f45f0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.416+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e81f4bb0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.417+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e80689e0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.417+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8e401b070 con 0x7fd8e80d15e0 2024-04-15T15:16:26.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.417+0000 7fd8dd7fa640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd8d406c8f0 0x7fd8d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.418+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd8e408b4f0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.418+0000 7fd8dffff640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd8d406c8f0 0x7fd8d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.418+0000 7fd8dffff640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd8d406c8f0 0x7fd8d406edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8000f60 tx=0x7fd8d8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.422+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd8e40595b0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.612+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8e80c9300 con 0x7fd8e80d15e0 2024-04-15T15:16:26.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.613+0000 7fd8dd7fa640 1 -- 172.21.15.46:0/3287337596 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7fd8e4058eb0 con 0x7fd8e80d15e0 2024-04-15T15:16:26.615 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:26.615 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd8d406c8f0 msgr2=0x7fd8d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd8d406c8f0 0x7fd8d406edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd8d8000f60 tx=0x7fd8d8009040 comp rx=0 tx=0).stop 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 msgr2=0x7fd8e8154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 0x7fd8e8154430 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c780 tx=0x7fd8e400cc40 comp rx=0 tx=0).stop 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 shutdown_connections 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd8d406c8f0 0x7fd8d406edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8e815e790 0x7fd8e80cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd8e80d5150 0x7fd8e80cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 --2- 172.21.15.46:0/3287337596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8e80d15e0 0x7fd8e8154430 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 >> 172.21.15.46:0/3287337596 conn(0x7fd8e8067b40 msgr2=0x7fd8e80b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 shutdown_connections 2024-04-15T15:16:26.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.615+0000 7fd8eede8640 1 -- 172.21.15.46:0/3287337596 wait complete. 2024-04-15T15:16:26.628 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:26.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.734+0000 7f4f6a51c640 1 Processor -- start 2024-04-15T15:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.734+0000 7f4f6a51c640 1 -- start start 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f6a51c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 0x7f4f640b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f6a51c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f640d2480 con 0x7f4f6415e790 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f6a51c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f640d2760 con 0x7f4f640d3240 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f6a51c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4f640d2db0 con 0x7f4f640d1610 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f63fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f6415e790 0x7f4f640cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34964/0 (socket says 172.21.15.46:34964) 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f63fff640 1 -- 172.21.15.46:0/960966124 learned_addr learned my addr 172.21.15.46:0/960966124 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:26.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f68ab9640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 0x7f4f640b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662519461 0 0) 0x7f4f640d2480 con 0x7f4f6415e790 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f4c003520 con 0x7f4f6415e790 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3893000251 0 0) 0x7f4f4c003520 con 0x7f4f6415e790 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 msgr2=0x7f4f640b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 0x7f4f640b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 >> v1:172.21.15.78:6789/0 conn(0x7f4f640d3240 legacy=0x7f4f640d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f640d2a40 con 0x7f4f6415e790 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f4f54001690 con 0x7f4f6415e790 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 >> v1:172.21.15.46:6789/0 conn(0x7f4f6415e790 legacy=0x7f4f640cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 0x7f4f4c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f4c003f60 0x7f4f4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f4c004d40 0x7f4f4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f640d2db0 con 0x7f4f4c003f60 2024-04-15T15:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4c00baa0 con 0x7f4f4c004d40 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.735+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4f4c00bca0 con 0x7f4f640d1610 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f637fe640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f4c003f60 0x7f4f4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f63fff640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 0x7f4f4c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f68ab9640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f4c004d40 0x7f4f4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f637fe640 1 -- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 msgr2=0x7f4f4c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f637fe640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f640d1610 0x7f4f4c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f637fe640 1 -- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f4c004d40 msgr2=0x7f4f4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f637fe640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f4c004d40 0x7f4f4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.736+0000 7f4f637fe640 1 -- 172.21.15.46:0/960966124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4c003520 con 0x7f4f4c003f60 2024-04-15T15:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.737+0000 7f4f637fe640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f4c003f60 0x7f4f4c004800 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4f58001750 tx=0x7f4f4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce989c55d0556ae server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.737+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58001370 con 0x7f4f4c003f60 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.737+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f580052d0 con 0x7f4f4c003f60 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.737+0000 7f4f62ffd640 1 -- 172.21.15.46:0/960966124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58003830 con 0x7f4f4c003f60 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f6a51c640 1 -- 172.21.15.46:0/960966124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f4c003f60 msgr2=0x7f4f4c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/960966124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f4c003f60 0x7f4f4c004800 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4f58001750 tx=0x7f4f4c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f68ab9640 1 -- 172.21.15.46:0/960966124 reap_dead start 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f6a51c640 1 -- 172.21.15.46:0/960966124 shutdown_connections 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f6a51c640 1 -- 172.21.15.46:0/960966124 >> 172.21.15.46:0/960966124 conn(0x7f4f6408bf60 msgr2=0x7f4f640d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f6a51c640 1 -- 172.21.15.46:0/960966124 shutdown_connections 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.738+0000 7f4f6a51c640 1 -- 172.21.15.46:0/960966124 wait complete. 2024-04-15T15:16:26.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.739+0000 7f4f6a51c640 1 Processor -- start 2024-04-15T15:16:26.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 -- start start 2024-04-15T15:16:26.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f64151b80 0x7f4f640c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f641f45a0 0x7f4f640c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 0x7f4f640ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f640d5180 con 0x7f4f640cc7c0 2024-04-15T15:16:26.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4f640d0430 con 0x7f4f64151b80 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f6a51c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4f640d5300 con 0x7f4f641f45a0 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f68ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 0x7f4f640ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f68ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 0x7f4f640ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37150/0 (socket says 172.21.15.46:37150) 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.740+0000 7f4f68ab9640 1 -- 172.21.15.46:0/3171224085 learned_addr learned my addr 172.21.15.46:0/3171224085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f637fe640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f641f45a0 0x7f4f640c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f63fff640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f64151b80 0x7f4f640c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f68ab9640 1 -- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f641f45a0 msgr2=0x7f4f640c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f68ab9640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f641f45a0 0x7f4f640c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f68ab9640 1 -- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f64151b80 msgr2=0x7f4f640c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f68ab9640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f64151b80 0x7f4f640c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f68ab9640 1 -- 172.21.15.46:0/3171224085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f6415fe60 con 0x7f4f640cc7c0 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f68ab9640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 0x7f4f640ccc40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4f640b7290 tx=0x7f4f5c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00fde0 con 0x7f4f640cc7c0 2024-04-15T15:16:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f5c01d070 con 0x7f4f640cc7c0 2024-04-15T15:16:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f64160140 con 0x7f4f640cc7c0 2024-04-15T15:16:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.741+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c015670 con 0x7f4f640cc7c0 2024-04-15T15:16:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.742+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f64160700 con 0x7f4f640cc7c0 2024-04-15T15:16:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.742+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f640cd180 con 0x7f4f640cc7c0 2024-04-15T15:16:26.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.743+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f5c015810 con 0x7f4f640cc7c0 2024-04-15T15:16:26.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.743+0000 7f4f60ff9640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f5006c920 0x7f4f5006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.743+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4f5c08c8e0 con 0x7f4f640cc7c0 2024-04-15T15:16:26.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.744+0000 7f4f63fff640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f5006c920 0x7f4f5006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.744+0000 7f4f63fff640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f5006c920 0x7f4f5006ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f640ba750 tx=0x7f4f54009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.748+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f5c010040 con 0x7f4f640cc7c0 2024-04-15T15:16:26.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.931+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f640689e0 con 0x7f4f640cc7c0 2024-04-15T15:16:26.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.931+0000 7f4f60ff9640 1 -- 172.21.15.46:0/3171224085 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f4f5c05a8f0 con 0x7f4f640cc7c0 2024-04-15T15:16:26.933 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:26.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f5006c920 msgr2=0x7f4f5006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f5006c920 0x7f4f5006ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4f640ba750 tx=0x7f4f54009040 comp rx=0 tx=0).stop 2024-04-15T15:16:26.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 msgr2=0x7f4f640ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 0x7f4f640ccc40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4f640b7290 tx=0x7f4f5c00f580 comp rx=0 tx=0).stop 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 shutdown_connections 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f5006c920 0x7f4f5006ede0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f640cc7c0 0x7f4f640ccc40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f641f45a0 0x7f4f640c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 --2- 172.21.15.46:0/3171224085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f64151b80 0x7f4f640c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 >> 172.21.15.46:0/3171224085 conn(0x7f4f6408bf60 msgr2=0x7f4f640d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 shutdown_connections 2024-04-15T15:16:26.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:26.933+0000 7f4f6a51c640 1 -- 172.21.15.46:0/3171224085 wait complete. 2024-04-15T15:16:26.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.052+0000 7f4a6a6e6640 1 Processor -- start 2024-04-15T15:16:27.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.052+0000 7f4a6a6e6640 1 -- start start 2024-04-15T15:16:27.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a6a6e6640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a640d15e0 0x7f4a640d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a6a6e6640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a6415eb70 con 0x7f4a640d4a40 2024-04-15T15:16:27.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a6a6e6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a640d3c80 con 0x7f4a6415e790 2024-04-15T15:16:27.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a6a6e6640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f4a640d3790 con 0x7f4a640d15e0 2024-04-15T15:16:27.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a68c83640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4a640d4a40 0x7f4a640d29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34972/0 (socket says 172.21.15.46:34972) 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a68c83640 1 -- 172.21.15.46:0/631474306 learned_addr learned my addr 172.21.15.46:0/631474306 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/631474306 >> v2:172.21.15.78:3300/0 conn(0x7f4a640d15e0 0x7f4a640d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1848553693 0 0) 0x7f4a6415eb70 con 0x7f4a640d4a40 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a50003520 con 0x7f4a640d4a40 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231002277 0 0) 0x7f4a640d3c80 con 0x7f4a6415e790 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a6415eb70 con 0x7f4a6415e790 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.053+0000 7f4a5bfff640 1 -- 172.21.15.46:0/631474306 >> v1:172.21.15.78:6789/0 conn(0x7f4a6415e790 legacy=0x7f4a640b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.054+0000 7f4a5bfff640 1 -- 172.21.15.46:0/631474306 >> v1:172.21.15.46:6789/0 conn(0x7f4a640d4a40 legacy=0x7f4a640d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.054+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2314808607 0 0) 0x7f4a50003520 con 0x7f4a640d4a40 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.054+0000 7f4a5bfff640 1 -- 172.21.15.46:0/631474306 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a640d3f60 con 0x7f4a640d15e0 2024-04-15T15:16:27.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.054+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/631474306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a640d1a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0097a0 tx=0x7f4a4c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75bdf2763c270c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c037070 con 0x7f4a640d15e0 2024-04-15T15:16:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a4c030510 con 0x7f4a640d15e0 2024-04-15T15:16:27.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a637fe640 1 -- 172.21.15.46:0/631474306 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c001a10 con 0x7f4a640d15e0 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/631474306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 msgr2=0x7f4a640d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/631474306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a640d1a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c0097a0 tx=0x7f4a4c02b710 comp rx=0 tx=0).stop 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/631474306 shutdown_connections 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/631474306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a640d1a90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/631474306 >> 172.21.15.46:0/631474306 conn(0x7f4a64067b40 msgr2=0x7f4a64067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/631474306 shutdown_connections 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.055+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/631474306 wait complete. 2024-04-15T15:16:27.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.056+0000 7f4a6a6e6640 1 Processor -- start 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.056+0000 7f4a6a6e6640 1 -- start start 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a6a6e6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a641f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a6a6e6640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640d4a40 0x7f4a640ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a6a6e6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 0x7f4a640b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a6a6e6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4a640b7db0 con 0x7f4a6415e790 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a6a6e6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a640d45b0 con 0x7f4a640d15e0 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a6a6e6640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4a640d30d0 con 0x7f4a640d4a40 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 0x7f4a640b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 0x7f4a640b99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37156/0 (socket says 172.21.15.46:37156) 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 -- 172.21.15.46:0/4234020773 learned_addr learned my addr 172.21.15.46:0/4234020773 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a63fff640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a641f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640d4a40 0x7f4a640ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 -- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640d4a40 msgr2=0x7f4a640ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640d4a40 0x7f4a640ab980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 -- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 msgr2=0x7f4a641f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a641f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a68c83640 1 -- 172.21.15.46:0/4234020773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a641508e0 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.057+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640d4a40 0x7f4a640ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.058+0000 7f4a68c83640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 0x7f4a640b99e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00cb00 tx=0x7f4a5c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.058+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c018070 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.058+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a5c00a640 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.058+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a64150bc0 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.058+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c00aa30 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.058+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a64151180 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.059+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a640689e0 con 0x7f4a6415e790 2024-04-15T15:16:27.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.059+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a5c00abd0 con 0x7f4a6415e790 2024-04-15T15:16:27.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.060+0000 7f4a617fa640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.060+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4a5c08b750 con 0x7f4a6415e790 2024-04-15T15:16:27.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.060+0000 7f4a63fff640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.060+0000 7f4a63fff640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a540011e0 tx=0x7f4a54009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.064+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a5c059700 con 0x7f4a6415e790 2024-04-15T15:16:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.246+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a640b79d0 con 0x7f4a6415e790 2024-04-15T15:16:27.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.246+0000 7f4a617fa640 1 -- 172.21.15.46:0/4234020773 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f4a5c059000 con 0x7f4a6415e790 2024-04-15T15:16:27.248 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:27.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 msgr2=0x7f4a4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a540011e0 tx=0x7f4a54009040 comp rx=0 tx=0).stop 2024-04-15T15:16:27.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 msgr2=0x7f4a640b99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 0x7f4a640b99e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c00cb00 tx=0x7f4a5c00cf00 comp rx=0 tx=0).stop 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 shutdown_connections 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a6415e790 0x7f4a640b99e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640d4a40 0x7f4a640ab980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 --2- 172.21.15.46:0/4234020773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d15e0 0x7f4a641f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 >> 172.21.15.46:0/4234020773 conn(0x7f4a64067b40 msgr2=0x7f4a640b84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 shutdown_connections 2024-04-15T15:16:27.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.248+0000 7f4a6a6e6640 1 -- 172.21.15.46:0/4234020773 wait complete. 2024-04-15T15:16:27.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:27.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.367+0000 7f16a8dd4640 1 Processor -- start 2024-04-15T15:16:27.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.367+0000 7f16a8dd4640 1 -- start start 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a8dd4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f16a415e790 0x7f16a40b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a8dd4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16a40b7be0 0x7f16a40d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a8dd4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a415eb70 con 0x7f16a40d1b20 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a8dd4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f16a40d1f90 con 0x7f16a40b7be0 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a8dd4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f16a40d3790 con 0x7f16a415e790 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f1699d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f16a40b7be0 0x7f16a40d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f16a40d1b20 0x7f16a40d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34986/0 (socket says 172.21.15.46:34986) 2024-04-15T15:16:27.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a2d9d640 1 -- 172.21.15.46:0/196929308 learned_addr learned my addr 172.21.15.46:0/196929308 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f1699d9b640 1 --2- 172.21.15.46:0/196929308 >> v2:172.21.15.46:3300/0 conn(0x7f16a40b7be0 0x7f16a40d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37172/0 (socket says 172.21.15.46:37172) 2024-04-15T15:16:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f16a259c640 1 --2- 172.21.15.46:0/196929308 >> v2:172.21.15.78:3300/0 conn(0x7f16a415e790 0x7f16a40b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f1699d9b640 1 -- 172.21.15.46:0/196929308 >> v2:172.21.15.78:3300/0 conn(0x7f16a415e790 msgr2=0x7f16a40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f1699d9b640 1 --2- 172.21.15.46:0/196929308 >> v2:172.21.15.78:3300/0 conn(0x7f16a415e790 0x7f16a40b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f1699d9b640 1 -- 172.21.15.46:0/196929308 >> v1:172.21.15.46:6789/0 conn(0x7f16a40d1b20 legacy=0x7f16a40d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.368+0000 7f1699d9b640 1 -- 172.21.15.46:0/196929308 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a40d3c80 con 0x7f16a40b7be0 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.369+0000 7f1699d9b640 1 --2- 172.21.15.46:0/196929308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a40d15e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f16a40d56e0 tx=0x7f168c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8088e37333f7e7a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.369+0000 7f16a1d9b640 1 -- 172.21.15.46:0/196929308 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c038070 con 0x7f16a40b7be0 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.369+0000 7f16a1d9b640 1 -- 172.21.15.46:0/196929308 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f168c031710 con 0x7f16a40b7be0 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.369+0000 7f16a1d9b640 1 -- 172.21.15.46:0/196929308 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c031ae0 con 0x7f16a40b7be0 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 -- 172.21.15.46:0/196929308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 msgr2=0x7f16a40d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/196929308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a40d15e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f16a40d56e0 tx=0x7f168c02b7b0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 -- 172.21.15.46:0/196929308 shutdown_connections 2024-04-15T15:16:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/196929308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a40d15e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/196929308 >> v2:172.21.15.78:3300/0 conn(0x7f16a415e790 0x7f16a40b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 -- 172.21.15.46:0/196929308 >> 172.21.15.46:0/196929308 conn(0x7f16a4067b40 msgr2=0x7f16a4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 -- 172.21.15.46:0/196929308 shutdown_connections 2024-04-15T15:16:27.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.370+0000 7f16a8dd4640 1 -- 172.21.15.46:0/196929308 wait complete. 2024-04-15T15:16:27.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.371+0000 7f16a8dd4640 1 Processor -- start 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.371+0000 7f16a8dd4640 1 -- start start 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a8dd4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a415f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a8dd4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16a40d1b20 0x7f16a4154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a8dd4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f16a415e790 0x7f16a40cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a8dd4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16a40d4240 con 0x7f16a40b7be0 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a8dd4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f16a40d45b0 con 0x7f16a40d1b20 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a8dd4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f16a40d5150 con 0x7f16a415e790 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a415f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37184/0 (socket says 172.21.15.46:37184) 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 -- 172.21.15.46:0/951245765 learned_addr learned my addr 172.21.15.46:0/951245765 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f1699d9b640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16a40d1b20 0x7f16a4154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a2d9d640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f16a415e790 0x7f16a40cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 -- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f16a415e790 msgr2=0x7f16a40cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f16a415e790 0x7f16a40cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 -- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16a40d1b20 msgr2=0x7f16a4154660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16a40d1b20 0x7f16a4154660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 -- 172.21.15.46:0/951245765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a41f4ca0 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f1699d9b640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16a40d1b20 0x7f16a4154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.372+0000 7f16a259c640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a415f980 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f169400d5f0 tx=0x7f169400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.373+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940125f0 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.373+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1694019070 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.373+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a415eea0 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.373+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940129d0 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.373+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a40cddf0 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.374+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a40689e0 con 0x7f16a40b7be0 2024-04-15T15:16:27.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.374+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1694012b70 con 0x7f16a40b7be0 2024-04-15T15:16:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.375+0000 7f169b7fe640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f168406c8f0 0x7f168406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.375+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f169408c280 con 0x7f16a40b7be0 2024-04-15T15:16:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.375+0000 7f1699d9b640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f168406c8f0 0x7f168406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.375+0000 7f1699d9b640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f168406c8f0 0x7f168406edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f168c035000 tx=0x7f168c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.379+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f169405a340 con 0x7f16a40b7be0 2024-04-15T15:16:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.563+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16a4074f20 con 0x7f16a40b7be0 2024-04-15T15:16:27.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.564+0000 7f169b7fe640 1 -- 172.21.15.46:0/951245765 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f1694059c40 con 0x7f16a40b7be0 2024-04-15T15:16:27.566 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f168406c8f0 msgr2=0x7f168406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f168406c8f0 0x7f168406edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f168c035000 tx=0x7f168c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 msgr2=0x7f16a415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a415f980 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f169400d5f0 tx=0x7f169400dab0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 shutdown_connections 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f168406c8f0 0x7f168406edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f16a415e790 0x7f16a40cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16a40d1b20 0x7f16a4154660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 --2- 172.21.15.46:0/951245765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16a40b7be0 0x7f16a415f980 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 >> 172.21.15.46:0/951245765 conn(0x7f16a4067b40 msgr2=0x7f16a40b85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 shutdown_connections 2024-04-15T15:16:27.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.566+0000 7f16a8dd4640 1 -- 172.21.15.46:0/951245765 wait complete. 2024-04-15T15:16:27.579 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:27.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.684+0000 7f4b174e4640 1 Processor -- start 2024-04-15T15:16:27.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.684+0000 7f4b174e4640 1 -- start start 2024-04-15T15:16:27.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b174e4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b1015e700 0x7f4b100b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b174e4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d3270 0x7f4b100cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b174e4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b100b47a0 con 0x7f4b100d2360 2024-04-15T15:16:27.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b174e4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4b100b53b0 con 0x7f4b1015e700 2024-04-15T15:16:27.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b174e4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4b100b4a80 con 0x7f4b100d3270 2024-04-15T15:16:27.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b1015e700 0x7f4b100b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b1015e700 0x7f4b100b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37190/0 (socket says 172.21.15.46:37190) 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 -- 172.21.15.46:0/3297264430 learned_addr learned my addr 172.21.15.46:0/3297264430 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15a81640 1 --2- 172.21.15.46:0/3297264430 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d3270 0x7f4b100cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 -- 172.21.15.46:0/3297264430 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d3270 msgr2=0x7f4b100cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 --2- 172.21.15.46:0/3297264430 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d3270 0x7f4b100cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 -- 172.21.15.46:0/3297264430 >> v1:172.21.15.78:6789/0 conn(0x7f4b100d2360 legacy=0x7f4b100d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.685+0000 7f4b15280640 1 -- 172.21.15.46:0/3297264430 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b100b4d60 con 0x7f4b1015e700 2024-04-15T15:16:27.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.686+0000 7f4b15280640 1 --2- 172.21.15.46:0/3297264430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b100b5840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009850 tx=0x7f4b0c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a04e3c9260f271 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.686+0000 7f4b07fff640 1 -- 172.21.15.46:0/3297264430 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c004230 con 0x7f4b1015e700 2024-04-15T15:16:27.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.686+0000 7f4b07fff640 1 -- 172.21.15.46:0/3297264430 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b0c0043d0 con 0x7f4b1015e700 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.686+0000 7f4b07fff640 1 -- 172.21.15.46:0/3297264430 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c003810 con 0x7f4b1015e700 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 -- 172.21.15.46:0/3297264430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 msgr2=0x7f4b100b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 --2- 172.21.15.46:0/3297264430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b100b5840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009850 tx=0x7f4b0c02b960 comp rx=0 tx=0).stop 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 -- 172.21.15.46:0/3297264430 shutdown_connections 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 --2- 172.21.15.46:0/3297264430 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d3270 0x7f4b100cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 --2- 172.21.15.46:0/3297264430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b100b5840 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 -- 172.21.15.46:0/3297264430 >> 172.21.15.46:0/3297264430 conn(0x7f4b1008bf60 msgr2=0x7f4b100d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 -- 172.21.15.46:0/3297264430 shutdown_connections 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.687+0000 7f4b174e4640 1 -- 172.21.15.46:0/3297264430 wait complete. 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.688+0000 7f4b174e4640 1 Processor -- start 2024-04-15T15:16:27.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.688+0000 7f4b174e4640 1 -- start start 2024-04-15T15:16:27.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b174e4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d2360 0x7f4b10161520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b174e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b100d3270 0x7f4b1015c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b174e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b1015cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b174e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b100b5d80 con 0x7f4b1015e700 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b174e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4b100b4450 con 0x7f4b100d3270 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b174e4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4b100d1d40 con 0x7f4b100d2360 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b1015cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b1015cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37206/0 (socket says 172.21.15.46:37206) 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 -- 172.21.15.46:0/1461822269 learned_addr learned my addr 172.21.15.46:0/1461822269 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b14a7f640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b100d3270 0x7f4b1015c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15280640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d2360 0x7f4b10161520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 -- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d2360 msgr2=0x7f4b10161520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d2360 0x7f4b10161520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 -- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b100d3270 msgr2=0x7f4b1015c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b100d3270 0x7f4b1015c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 -- 172.21.15.46:0/1461822269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b100b3da0 con 0x7f4b1015e700 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b15a81640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b1015cd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007e70 tx=0x7f4b0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08014420 con 0x7f4b1015e700 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b08019070 con 0x7f4b1015e700 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.689+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08014820 con 0x7f4b1015e700 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.690+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1015ec10 con 0x7f4b1015e700 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.690+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b1015f1d0 con 0x7f4b1015e700 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.691+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1015f3d0 con 0x7f4b1015e700 2024-04-15T15:16:27.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.691+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b080149c0 con 0x7f4b1015e700 2024-04-15T15:16:27.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.691+0000 7f4b05ffb640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4af406c8f0 0x7f4af406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.692+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4b0808c880 con 0x7f4b1015e700 2024-04-15T15:16:27.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.692+0000 7f4b15280640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4af406c8f0 0x7f4af406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.692+0000 7f4b15280640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4af406c8f0 0x7f4af406edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c035000 tx=0x7f4b0c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.696+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b1015f3d0 con 0x7f4b1015e700 2024-04-15T15:16:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.881+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4b100689e0 con 0x7f4b1015e700 2024-04-15T15:16:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.881+0000 7f4b05ffb640 1 -- 172.21.15.46:0/1461822269 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f4b0805a8b0 con 0x7f4b1015e700 2024-04-15T15:16:27.883 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4af406c8f0 msgr2=0x7f4af406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4af406c8f0 0x7f4af406edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c035000 tx=0x7f4b0c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 msgr2=0x7f4b1015cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b1015cd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4b08007e70 tx=0x7f4b0800e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 shutdown_connections 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4af406c8f0 0x7f4af406edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b1015e700 0x7f4b1015cd20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b100d3270 0x7f4b1015c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 --2- 172.21.15.46:0/1461822269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b100d2360 0x7f4b10161520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.883+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 >> 172.21.15.46:0/1461822269 conn(0x7f4b1008bf60 msgr2=0x7f4b1015ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.884+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 shutdown_connections 2024-04-15T15:16:27.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:27.884+0000 7f4b174e4640 1 -- 172.21.15.46:0/1461822269 wait complete. 2024-04-15T15:16:27.896 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:16:28.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.000+0000 7fb530bcd640 1 Processor -- start 2024-04-15T15:16:28.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.000+0000 7fb530bcd640 1 -- start start 2024-04-15T15:16:28.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb530bcd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c15ec10 0x7fb52c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb530bcd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb52c0b53c0 0x7fb52c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb530bcd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb52c067e30 con 0x7fb52c15e830 2024-04-15T15:16:28.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb530bcd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb52c0b3c10 con 0x7fb52c0b53c0 2024-04-15T15:16:28.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb530bcd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c0b3ef0 con 0x7fb52c15ec10 2024-04-15T15:16:28.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb52ad9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb52c0b53c0 0x7fb52c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb523fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c15ec10 0x7fb52c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb52a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb52c15e830 0x7fb52c067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:56012/0 (socket says 172.21.15.46:56012) 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb52a59c640 1 -- 172.21.15.46:0/1999335930 learned_addr learned my addr 172.21.15.46:0/1999335930 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb529d9b640 1 -- 172.21.15.46:0/1999335930 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1287275161 0 0) 0x7fb52c067e30 con 0x7fb52c15e830 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb529d9b640 1 -- 172.21.15.46:0/1999335930 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb518003520 con 0x7fb52c15e830 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb523fff640 1 -- 172.21.15.46:0/1999335930 >> v2:172.21.15.78:3300/0 conn(0x7fb52c0b53c0 msgr2=0x7fb52c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb523fff640 1 --2- 172.21.15.46:0/1999335930 >> v2:172.21.15.78:3300/0 conn(0x7fb52c0b53c0 0x7fb52c0b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb523fff640 1 -- 172.21.15.46:0/1999335930 >> v1:172.21.15.78:6789/0 conn(0x7fb52c15e830 legacy=0x7fb52c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb523fff640 1 -- 172.21.15.46:0/1999335930 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c0b41d0 con 0x7fb52c15ec10 2024-04-15T15:16:28.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.001+0000 7fb52ad9d640 1 --2- 172.21.15.46:0/1999335930 >> v2:172.21.15.78:3300/0 conn(0x7fb52c0b53c0 0x7fb52c0b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.002+0000 7fb523fff640 1 --2- 172.21.15.46:0/1999335930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c15ec10 0x7fb52c0b4cb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5140097a0 tx=0x7fb51402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5dce4b5fae5f5ef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.002+0000 7fb529d9b640 1 -- 172.21.15.46:0/1999335930 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514035020 con 0x7fb52c15ec10 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.002+0000 7fb529d9b640 1 -- 172.21.15.46:0/1999335930 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb514031830 con 0x7fb52c15ec10 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.002+0000 7fb529d9b640 1 -- 172.21.15.46:0/1999335930 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514031be0 con 0x7fb52c15ec10 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 -- 172.21.15.46:0/1999335930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c15ec10 msgr2=0x7fb52c0b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 --2- 172.21.15.46:0/1999335930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c15ec10 0x7fb52c0b4cb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5140097a0 tx=0x7fb51402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 -- 172.21.15.46:0/1999335930 shutdown_connections 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 --2- 172.21.15.46:0/1999335930 >> v2:172.21.15.78:3300/0 conn(0x7fb52c0b53c0 0x7fb52c0b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 --2- 172.21.15.46:0/1999335930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c15ec10 0x7fb52c0b4cb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 -- 172.21.15.46:0/1999335930 >> 172.21.15.46:0/1999335930 conn(0x7fb52c073360 msgr2=0x7fb52c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 -- 172.21.15.46:0/1999335930 shutdown_connections 2024-04-15T15:16:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.003+0000 7fb530bcd640 1 -- 172.21.15.46:0/1999335930 wait complete. 2024-04-15T15:16:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.004+0000 7fb530bcd640 1 Processor -- start 2024-04-15T15:16:28.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.004+0000 7fb530bcd640 1 -- start start 2024-04-15T15:16:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb530bcd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb52c0b53c0 0x7fb52c145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb530bcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 0x7fb52c0c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb530bcd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c0c6020 0x7fb52c15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb530bcd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c15ff60 con 0x7fb52c15e830 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb530bcd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb52c0b4790 con 0x7fb52c0b53c0 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb530bcd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb52c1600e0 con 0x7fb52c0c6020 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 0x7fb52c0c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 0x7fb52c0c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37208/0 (socket says 172.21.15.46:37208) 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 -- 172.21.15.46:0/493226683 learned_addr learned my addr 172.21.15.46:0/493226683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb52a59c640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb52c0b53c0 0x7fb52c145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb52ad9d640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c0c6020 0x7fb52c15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 -- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c0c6020 msgr2=0x7fb52c15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c0c6020 0x7fb52c15c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 -- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb52c0b53c0 msgr2=0x7fb52c145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb52c0b53c0 0x7fb52c145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.005+0000 7fb523fff640 1 -- 172.21.15.46:0/493226683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52c146120 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.006+0000 7fb523fff640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 0x7fb52c0c5ae0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb514000c00 tx=0x7fb5140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.006+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514040070 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.006+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb51403a830 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.006+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52c146390 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.006+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514035040 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.006+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52c154470 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.007+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb52c0c6430 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.007+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb51404b060 con 0x7fb52c15e830 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.008+0000 7fb522ffd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb50c06c920 0x7fb50c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.008+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb5140ad590 con 0x7fb52c15e830 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.008+0000 7fb52a59c640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb50c06c920 0x7fb50c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.009+0000 7fb52a59c640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb50c06c920 0x7fb50c06ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005880 tx=0x7fb51c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.012+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb52c0c6430 con 0x7fb52c15e830 2024-04-15T15:16:28.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.268+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7fb52c06ff10 con 0x7fb52c15e830 2024-04-15T15:16:28.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.748+0000 7fb522ffd640 1 -- 172.21.15.46:0/493226683 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v70) ==== 140+0+0 (secure 0 0 0) 0x7fb51407b5c0 con 0x7fb52c15e830 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb50c06c920 msgr2=0x7fb50c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb50c06c920 0x7fb50c06ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb51c005880 tx=0x7fb51c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 msgr2=0x7fb52c0c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 0x7fb52c0c5ae0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb514000c00 tx=0x7fb5140046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 shutdown_connections 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb50c06c920 0x7fb50c06ede0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb52c0c6020 0x7fb52c15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb52c15e830 0x7fb52c0c5ae0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 --2- 172.21.15.46:0/493226683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb52c0b53c0 0x7fb52c145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 >> 172.21.15.46:0/493226683 conn(0x7fb52c073360 msgr2=0x7fb52c0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 shutdown_connections 2024-04-15T15:16:28.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.751+0000 7fb530bcd640 1 -- 172.21.15.46:0/493226683 wait complete. 2024-04-15T15:16:28.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-15T15:16:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.875+0000 7fe38144a640 1 Processor -- start 2024-04-15T15:16:28.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.875+0000 7fe38144a640 1 -- start start 2024-04-15T15:16:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe38144a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe37c15e790 0x7fe37c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe38144a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 0x7fe37c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe38144a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37c0d2480 con 0x7fe37c0cfd20 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe38144a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe37c0d2db0 con 0x7fe37c15e790 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe38144a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe37c0d2760 con 0x7fe37c0d1610 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe37b7fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 0x7fe37c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe37affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe37c15e790 0x7fe37c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe37a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe37c0cfd20 0x7fe37c0d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:56016/0 (socket says 172.21.15.46:56016) 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe37b7fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 0x7fe37c0d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:36942/0 (socket says 172.21.15.46:36942) 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe37a7fc640 1 -- 172.21.15.46:0/2296848901 learned_addr learned my addr 172.21.15.46:0/2296848901 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.876+0000 7fe37affd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe37c15e790 0x7fe37c0d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:41074/0 (socket says 172.21.15.46:41074) 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe37affd640 1 -- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 msgr2=0x7fe37c0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe37affd640 1 --2- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 0x7fe37c0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe37affd640 1 -- 172.21.15.46:0/2296848901 >> v1:172.21.15.78:6789/0 conn(0x7fe37c0cfd20 legacy=0x7fe37c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe37affd640 1 -- 172.21.15.46:0/2296848901 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c0d2a40 con 0x7fe37c15e790 2024-04-15T15:16:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe379ffb640 1 -- 172.21.15.46:0/2296848901 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2259625239 0 0) 0x7fe37c0d2480 con 0x7fe37c0cfd20 2024-04-15T15:16:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe37b7fe640 1 --2- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 0x7fe37c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.877+0000 7fe37affd640 1 --2- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c15e790 0x7fe37c0d3240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0d2f30 tx=0x7fe36c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d248953c6d5c812d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe379ffb640 1 -- 172.21.15.46:0/2296848901 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c035020 con 0x7fe37c15e790 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe379ffb640 1 -- 172.21.15.46:0/2296848901 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c031800 con 0x7fe37c15e790 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe379ffb640 1 -- 172.21.15.46:0/2296848901 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c031bd0 con 0x7fe37c15e790 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 -- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c15e790 msgr2=0x7fe37c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 --2- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c15e790 0x7fe37c0d3240 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0d2f30 tx=0x7fe36c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 -- 172.21.15.46:0/2296848901 shutdown_connections 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 --2- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0d1610 0x7fe37c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 --2- 172.21.15.46:0/2296848901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c15e790 0x7fe37c0d3240 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 -- 172.21.15.46:0/2296848901 >> 172.21.15.46:0/2296848901 conn(0x7fe37c08bf60 msgr2=0x7fe37c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 -- 172.21.15.46:0/2296848901 shutdown_connections 2024-04-15T15:16:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.878+0000 7fe38144a640 1 -- 172.21.15.46:0/2296848901 wait complete. 2024-04-15T15:16:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.879+0000 7fe38144a640 1 Processor -- start 2024-04-15T15:16:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.879+0000 7fe38144a640 1 -- start start 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe38144a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 0x7fe37c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe38144a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d1610 0x7fe37c09e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe38144a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c15e790 0x7fe37c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe38144a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c15ecf0 con 0x7fe37c0cfd20 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe38144a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c15eb70 con 0x7fe37c0d1610 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe38144a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe37c0b74f0 con 0x7fe37c15e790 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe37affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 0x7fe37c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe37affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 0x7fe37c15b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37220/0 (socket says 172.21.15.46:37220) 2024-04-15T15:16:28.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe37affd640 1 -- 172.21.15.46:0/1158209106 learned_addr learned my addr 172.21.15.46:0/1158209106 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe37a7fc640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d1610 0x7fe37c09e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.880+0000 7fe37b7fe640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c15e790 0x7fe37c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe37affd640 1 -- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c15e790 msgr2=0x7fe37c15bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe37affd640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c15e790 0x7fe37c15bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe37affd640 1 -- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d1610 msgr2=0x7fe37c09e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe37affd640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d1610 0x7fe37c09e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe37affd640 1 -- 172.21.15.46:0/1158209106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c0a0360 con 0x7fe37c0cfd20 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe37affd640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 0x7fe37c15b310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe36c000c00 tx=0x7fe36c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c040070 con 0x7fe37c0cfd20 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c039cf0 con 0x7fe37c0cfd20 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c035040 con 0x7fe37c0cfd20 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c0a0640 con 0x7fe37c0cfd20 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.881+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c1f7e30 con 0x7fe37c0cfd20 2024-04-15T15:16:28.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.882+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37c0689e0 con 0x7fe37c0cfd20 2024-04-15T15:16:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.882+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe36c04b060 con 0x7fe37c0cfd20 2024-04-15T15:16:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.883+0000 7fe3737fe640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35c06c920 0x7fe35c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.883+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe36c0ad980 con 0x7fe37c0cfd20 2024-04-15T15:16:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.883+0000 7fe37a7fc640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35c06c920 0x7fe35c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.884+0000 7fe37a7fc640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35c06c920 0x7fe35c06ede0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0d0af0 tx=0x7fe364007b90 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:28.887+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe36c07ba40 con 0x7fe37c0cfd20 2024-04-15T15:16:29.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.071+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe37c0d1ce0 con 0x7fe37c0cfd20 2024-04-15T15:16:29.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.074+0000 7fe3737fe640 1 -- 172.21.15.46:0/1158209106 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fe36c07b340 con 0x7fe37c0cfd20 2024-04-15T15:16:29.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.076+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35c06c920 msgr2=0x7fe35c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.076+0000 7fe38144a640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35c06c920 0x7fe35c06ede0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe37c0d0af0 tx=0x7fe364007b90 comp rx=0 tx=0).stop 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.076+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 msgr2=0x7fe37c15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.076+0000 7fe38144a640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 0x7fe37c15b310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe36c000c00 tx=0x7fe36c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.076+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 shutdown_connections 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.076+0000 7fe38144a640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe35c06c920 0x7fe35c06ede0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.077+0000 7fe38144a640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c15e790 0x7fe37c15bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.077+0000 7fe38144a640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d1610 0x7fe37c09e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.077+0000 7fe38144a640 1 --2- 172.21.15.46:0/1158209106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0cfd20 0x7fe37c15b310 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.077+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 >> 172.21.15.46:0/1158209106 conn(0x7fe37c08bf60 msgr2=0x7fe37c074d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.077+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 shutdown_connections 2024-04-15T15:16:29.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.077+0000 7fe38144a640 1 -- 172.21.15.46:0/1158209106 wait complete. 2024-04-15T15:16:29.089 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:16:29.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:16:29.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:16:29.195 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:16:29.196 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:16:29.196 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:29.211 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:16:29.212 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:29.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:29.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.411+0000 7f2f83847640 1 Processor -- start 2024-04-15T15:16:29.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.411+0000 7f2f83847640 1 -- start start 2024-04-15T15:16:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f83847640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f7c0701f0 0x7f2f7c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f83847640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c15fb30 0x7f2f7c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f83847640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f7c070720 con 0x7f2f7c09a9d0 2024-04-15T15:16:29.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f83847640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2f7c09ba90 con 0x7f2f7c0701f0 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f83847640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2f7c0d08c0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f80de2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f7c0701f0 0x7f2f7c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f80de2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f7c0701f0 0x7f2f7c160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37232/0 (socket says 172.21.15.46:37232) 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f80de2640 1 -- 172.21.15.46:0/1096234035 learned_addr learned my addr 172.21.15.46:0/1096234035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f81de4640 1 --2- 172.21.15.46:0/1096234035 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c15fb30 0x7f2f7c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f80de2640 1 -- 172.21.15.46:0/1096234035 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c15fb30 msgr2=0x7f2f7c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f80de2640 1 --2- 172.21.15.46:0/1096234035 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c15fb30 0x7f2f7c0d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.412+0000 7f2f80de2640 1 -- 172.21.15.46:0/1096234035 >> v1:172.21.15.78:6789/0 conn(0x7f2f7c09a9d0 legacy=0x7f2f7c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.413+0000 7f2f80de2640 1 -- 172.21.15.46:0/1096234035 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c070a90 con 0x7f2f7c0701f0 2024-04-15T15:16:29.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.413+0000 7f2f80de2640 1 --2- 172.21.15.46:0/1096234035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c160420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009870 tx=0x7f2f7002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d9f6f2285fed5c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.413+0000 7f2f7bfff640 1 -- 172.21.15.46:0/1096234035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70034070 con 0x7f2f7c0701f0 2024-04-15T15:16:29.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.413+0000 7f2f7bfff640 1 -- 172.21.15.46:0/1096234035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f70030840 con 0x7f2f7c0701f0 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.413+0000 7f2f7bfff640 1 -- 172.21.15.46:0/1096234035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70030c10 con 0x7f2f7c0701f0 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 -- 172.21.15.46:0/1096234035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c0701f0 msgr2=0x7f2f7c160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 --2- 172.21.15.46:0/1096234035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c160420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009870 tx=0x7f2f7002b980 comp rx=0 tx=0).stop 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 -- 172.21.15.46:0/1096234035 shutdown_connections 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 --2- 172.21.15.46:0/1096234035 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c15fb30 0x7f2f7c0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 --2- 172.21.15.46:0/1096234035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c160420 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 -- 172.21.15.46:0/1096234035 >> 172.21.15.46:0/1096234035 conn(0x7f2f7c0d2460 msgr2=0x7f2f7c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 -- 172.21.15.46:0/1096234035 shutdown_connections 2024-04-15T15:16:29.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.414+0000 7f2f83847640 1 -- 172.21.15.46:0/1096234035 wait complete. 2024-04-15T15:16:29.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.415+0000 7f2f83847640 1 Processor -- start 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.415+0000 7f2f83847640 1 -- start start 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f83847640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f83847640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c09a9d0 0x7f2f7c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f83847640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 0x7f2f7c161a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f83847640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c15ffc0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f83847640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c071050 con 0x7f2f7c0701f0 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f83847640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2f7c0b1c60 con 0x7f2f7c09a9d0 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f81de4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 0x7f2f7c161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f81de4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 0x7f2f7c161a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37246/0 (socket says 172.21.15.46:37246) 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f81de4640 1 -- 172.21.15.46:0/1993077599 learned_addr learned my addr 172.21.15.46:0/1993077599 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.416+0000 7f2f81de4640 1 -- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c09a9d0 msgr2=0x7f2f7c0b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f80de2640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c09a9d0 0x7f2f7c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f815e3640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f81de4640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c09a9d0 0x7f2f7c0b5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f81de4640 1 -- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f7c0701f0 msgr2=0x7f2f7c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f81de4640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c1549b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f81de4640 1 -- 172.21.15.46:0/1993077599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c1f4f80 con 0x7f2f7c15fb30 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f81de4640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 0x7f2f7c161a60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007e60 tx=0x7f2f6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c014420 con 0x7f2f7c15fb30 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f6c019070 con 0x7f2f7c15fb30 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c0b7970 con 0x7f2f7c15fb30 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c014820 con 0x7f2f7c15fb30 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.417+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c0b7f30 con 0x7f2f7c15fb30 2024-04-15T15:16:29.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.418+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7c0689e0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.419+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f6c0149c0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.419+0000 7f2f79ffb640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f6806c8f0 0x7f2f6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.419+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2f6c08c7b0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.419+0000 7f2f815e3640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f6806c8f0 0x7f2f6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.420+0000 7f2f815e3640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f6806c8f0 0x7f2f6806edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c06db30 tx=0x7f2f74009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.423+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f6c05a7e0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.609+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f7c0c8730 con 0x7f2f7c15fb30 2024-04-15T15:16:29.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.610+0000 7f2f79ffb640 1 -- 172.21.15.46:0/1993077599 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f2f6c05a0e0 con 0x7f2f7c15fb30 2024-04-15T15:16:29.612 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:29.612 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f6806c8f0 msgr2=0x7f2f6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f6806c8f0 0x7f2f6806edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c06db30 tx=0x7f2f74009040 comp rx=0 tx=0).stop 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 msgr2=0x7f2f7c161a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 0x7f2f7c161a60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007e60 tx=0x7f2f6c00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 shutdown_connections 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2f6806c8f0 0x7f2f6806edb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f7c15fb30 0x7f2f7c161a60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2f7c09a9d0 0x7f2f7c0b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 --2- 172.21.15.46:0/1993077599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2f7c0701f0 0x7f2f7c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 >> 172.21.15.46:0/1993077599 conn(0x7f2f7c0d2460 msgr2=0x7f2f7c06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 shutdown_connections 2024-04-15T15:16:29.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.612+0000 7f2f83847640 1 -- 172.21.15.46:0/1993077599 wait complete. 2024-04-15T15:16:29.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:29.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.732+0000 7f36b5216640 1 Processor -- start 2024-04-15T15:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.732+0000 7f36b5216640 1 -- start start 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36b5216640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f36b015e790 0x7f36b00d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36b5216640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b00d2480 con 0x7f36b00cfd20 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36b5216640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b00d2760 con 0x7f36b00d1610 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36b5216640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f36b00d2db0 con 0x7f36b015e790 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36a7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36b00cfd20 0x7f36b00d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32956/0 (socket says 172.21.15.46:32956) 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36a7fff640 1 -- 172.21.15.46:0/3711220659 learned_addr learned my addr 172.21.15.46:0/3711220659 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3711220659 >> v2:172.21.15.78:3300/0 conn(0x7f36b015e790 0x7f36b00d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3103621430 0 0) 0x7f36b00d2480 con 0x7f36b00cfd20 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f369c003520 con 0x7f36b00cfd20 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 11892714 0 0) 0x7f369c003520 con 0x7f36b00cfd20 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 >> v2:172.21.15.78:3300/0 conn(0x7f36b015e790 msgr2=0x7f36b00d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 --2- 172.21.15.46:0/3711220659 >> v2:172.21.15.78:3300/0 conn(0x7f36b015e790 0x7f36b00d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 >> v1:172.21.15.78:6789/0 conn(0x7f36b00d1610 legacy=0x7f36b00b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b00d2a40 con 0x7f36b00cfd20 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f3698001670 con 0x7f36b00cfd20 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 >> v1:172.21.15.46:6789/0 conn(0x7f36b00cfd20 legacy=0x7f36b00d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f369c0039b0 0x7f369c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f369c0043d0 0x7f369c0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0050a0 0x7f369c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f36b00d2db0 con 0x7f369c0039b0 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f369c00bfe0 con 0x7f369c0043d0 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f369c00c1e0 con 0x7f369c0050a0 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.733+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f369c0039b0 0x7f369c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36a7fff640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f369c0043d0 0x7f369c0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36af59e640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0050a0 0x7f369c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36aed9d640 1 -- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0050a0 msgr2=0x7f369c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0050a0 0x7f369c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36aed9d640 1 -- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f369c0043d0 msgr2=0x7f369c0048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f369c0043d0 0x7f369c0048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36aed9d640 1 -- 172.21.15.46:0/3711220659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c003520 con 0x7f369c0039b0 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36a7fff640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f369c0043d0 0x7f369c0048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36af59e640 1 -- 172.21.15.46:0/3711220659 reap_dead start 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.734+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f369c0039b0 0x7f369c003db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009ac0 tx=0x7f369c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d57aff61a9161d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0039070 con 0x7f369c0039b0 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a002c6b0 con 0x7f369c0039b0 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36ae59c640 1 -- 172.21.15.46:0/3711220659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0009690 con 0x7f369c0039b0 2024-04-15T15:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 -- 172.21.15.46:0/3711220659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f369c0039b0 msgr2=0x7f369c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f369c0039b0 0x7f369c003db0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009ac0 tx=0x7f369c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 -- 172.21.15.46:0/3711220659 shutdown_connections 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 --2- 172.21.15.46:0/3711220659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f369c0039b0 0x7f369c003db0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 -- 172.21.15.46:0/3711220659 >> 172.21.15.46:0/3711220659 conn(0x7f36b008bf60 msgr2=0x7f36b00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 -- 172.21.15.46:0/3711220659 shutdown_connections 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.735+0000 7f36b5216640 1 -- 172.21.15.46:0/3711220659 wait complete. 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 Processor -- start 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 -- start start 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0039b0 0x7f36b00cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36b009e400 0x7f36b00cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 0x7f36b00cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f36b015ecf0 con 0x7f36b00ccd00 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f36b015eb70 con 0x7f36b009e400 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36b5216640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f36b00d0160 con 0x7f369c0039b0 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36af59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 0x7f36b00cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36af59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 0x7f36b00cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43736/0 (socket says 172.21.15.46:43736) 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.737+0000 7f36af59e640 1 -- 172.21.15.46:0/3727421652 learned_addr learned my addr 172.21.15.46:0/3727421652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0039b0 0x7f36b00cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36a7fff640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36b009e400 0x7f36b00cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36af59e640 1 -- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0039b0 msgr2=0x7f36b00cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36af59e640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0039b0 0x7f36b00cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36af59e640 1 -- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36b009e400 msgr2=0x7f36b00cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36af59e640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36b009e400 0x7f36b00cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36af59e640 1 -- 172.21.15.46:0/3727421652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b009e8c0 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36af59e640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 0x7f36b00cd1b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f36b00d2d20 tx=0x7f36a800ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a8003b80 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a8018070 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b014dc40 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a80050d0 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.738+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b014e200 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.739+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b014e400 con 0x7f36b00ccd00 2024-04-15T15:16:29.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.740+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36a8005270 con 0x7f36b00ccd00 2024-04-15T15:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.740+0000 7f36a77fe640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f369006c920 0x7f369006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.740+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f36a808bb80 con 0x7f36b00ccd00 2024-04-15T15:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.740+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f369006c920 0x7f369006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.741+0000 7f36aed9d640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f369006c920 0x7f369006ede0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f36a0036000 tx=0x7f36a0030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.745+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36b014e400 con 0x7f36b00ccd00 2024-04-15T15:16:29.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.932+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36b00689e0 con 0x7f36b00ccd00 2024-04-15T15:16:29.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.933+0000 7f36a77fe640 1 -- 172.21.15.46:0/3727421652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f36a8059bb0 con 0x7f36b00ccd00 2024-04-15T15:16:29.935 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f369006c920 msgr2=0x7f369006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f369006c920 0x7f369006ede0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f36a0036000 tx=0x7f36a0030000 comp rx=0 tx=0).stop 2024-04-15T15:16:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 msgr2=0x7f36b00cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 0x7f36b00cd1b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f36b00d2d20 tx=0x7f36a800ead0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 shutdown_connections 2024-04-15T15:16:29.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f369006c920 0x7f369006ede0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36b00ccd00 0x7f36b00cd1b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36b009e400 0x7f36b00cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 --2- 172.21.15.46:0/3727421652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f369c0039b0 0x7f36b00cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 >> 172.21.15.46:0/3727421652 conn(0x7f36b008bf60 msgr2=0x7f36b006e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 shutdown_connections 2024-04-15T15:16:29.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:29.935+0000 7f36b5216640 1 -- 172.21.15.46:0/3727421652 wait complete. 2024-04-15T15:16:29.948 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:30.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 Processor -- start 2024-04-15T15:16:30.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 -- start start 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ea40 0x7f943c0b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f943c15ee20 0x7f943c0b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f943c0cff70 con 0x7f943c0b65c0 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f943c0b6a10 con 0x7f943c15ee20 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.054+0000 7f94409e9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f943c0d0250 con 0x7f943c15ea40 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f9433fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f943c15ee20 0x7f943c0b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f943a59c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ea40 0x7f943c0b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f943ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f943c0b65c0 0x7f943c0d2470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:41392/0 (socket says 172.21.15.46:41392) 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f943ad9d640 1 -- 172.21.15.46:0/1609317898 learned_addr learned my addr 172.21.15.46:0/1609317898 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f9433fff640 1 -- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ea40 msgr2=0x7f943c0b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f9433fff640 1 --2- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ea40 0x7f943c0b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f9433fff640 1 -- 172.21.15.46:0/1609317898 >> v1:172.21.15.78:6789/0 conn(0x7f943c0b65c0 legacy=0x7f943c0d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f9439d9b640 1 -- 172.21.15.46:0/1609317898 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481837198 0 0) 0x7f943c0cff70 con 0x7f943c0b65c0 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f9433fff640 1 -- 172.21.15.46:0/1609317898 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943c0d0530 con 0x7f943c15ee20 2024-04-15T15:16:30.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.055+0000 7f943a59c640 1 --2- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ea40 0x7f943c0b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.056+0000 7f9433fff640 1 --2- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ee20 0x7f943c0b6080 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94240097a0 tx=0x7f942402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23532c8ec5d87439 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.056+0000 7f9439d9b640 1 -- 172.21.15.46:0/1609317898 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424034070 con 0x7f943c15ee20 2024-04-15T15:16:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.056+0000 7f9439d9b640 1 -- 172.21.15.46:0/1609317898 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9424030610 con 0x7f943c15ee20 2024-04-15T15:16:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.056+0000 7f9439d9b640 1 -- 172.21.15.46:0/1609317898 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94240309e0 con 0x7f943c15ee20 2024-04-15T15:16:30.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 -- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ee20 msgr2=0x7f943c0b6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 --2- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ee20 0x7f943c0b6080 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94240097a0 tx=0x7f942402b960 comp rx=0 tx=0).stop 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 -- 172.21.15.46:0/1609317898 shutdown_connections 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 --2- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ee20 0x7f943c0b6080 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 --2- 172.21.15.46:0/1609317898 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ea40 0x7f943c0b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 -- 172.21.15.46:0/1609317898 >> 172.21.15.46:0/1609317898 conn(0x7f943c08bf60 msgr2=0x7f943c0d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 -- 172.21.15.46:0/1609317898 shutdown_connections 2024-04-15T15:16:30.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.057+0000 7f94409e9640 1 -- 172.21.15.46:0/1609317898 wait complete. 2024-04-15T15:16:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.058+0000 7f94409e9640 1 Processor -- start 2024-04-15T15:16:30.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.058+0000 7f94409e9640 1 -- start start 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f94409e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 0x7f943c0b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f94409e9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ea40 0x7f943c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f94409e9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ee20 0x7f943c153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f94409e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f943c0b38c0 con 0x7f943c0b65c0 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f94409e9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f943c0d0af0 con 0x7f943c15ea40 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f94409e9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f943c0b3a40 con 0x7f943c15ee20 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 0x7f943c0b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 0x7f943c0b82b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43744/0 (socket says 172.21.15.46:43744) 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 -- 172.21.15.46:0/3371301634 learned_addr learned my addr 172.21.15.46:0/3371301634 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f9433fff640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ea40 0x7f943c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943ad9d640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ee20 0x7f943c153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 -- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ee20 msgr2=0x7f943c153a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ee20 0x7f943c153a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 -- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ea40 msgr2=0x7f943c0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ea40 0x7f943c0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.059+0000 7f943a59c640 1 -- 172.21.15.46:0/3371301634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943c0b74f0 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.060+0000 7f943a59c640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 0x7f943c0b82b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f942c00ebe0 tx=0x7f942c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.060+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c017070 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.060+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942c00cc30 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.060+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943c0b77d0 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.060+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942c013470 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.060+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f943c0ce180 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.061+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f943c0689e0 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.061+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f942c01a070 con 0x7f943c0b65c0 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.062+0000 7f9432ffd640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f941c06c8f0 0x7f941c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.062+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f942c08c130 con 0x7f943c0b65c0 2024-04-15T15:16:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.062+0000 7f9433fff640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f941c06c8f0 0x7f941c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.063+0000 7f9433fff640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f941c06c8f0 0x7f941c06edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f942403b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.066+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f942c059ba0 con 0x7f943c0b65c0 2024-04-15T15:16:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.249+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f943c0d4f00 con 0x7f943c0b65c0 2024-04-15T15:16:30.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.250+0000 7f9432ffd640 1 -- 172.21.15.46:0/3371301634 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f942c05dea0 con 0x7f943c0b65c0 2024-04-15T15:16:30.252 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f941c06c8f0 msgr2=0x7f941c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f941c06c8f0 0x7f941c06edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f942403b040 comp rx=0 tx=0).stop 2024-04-15T15:16:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 msgr2=0x7f943c0b82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 0x7f943c0b82b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f942c00ebe0 tx=0x7f942c00c520 comp rx=0 tx=0).stop 2024-04-15T15:16:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 shutdown_connections 2024-04-15T15:16:30.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f941c06c8f0 0x7f941c06edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f943c15ee20 0x7f943c153a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f943c15ea40 0x7f943c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 --2- 172.21.15.46:0/3371301634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f943c0b65c0 0x7f943c0b82b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 >> 172.21.15.46:0/3371301634 conn(0x7f943c08bf60 msgr2=0x7f943c0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 shutdown_connections 2024-04-15T15:16:30.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.252+0000 7f94409e9640 1 -- 172.21.15.46:0/3371301634 wait complete. 2024-04-15T15:16:30.265 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:30.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.370+0000 7f4d27a7e640 1 Processor -- start 2024-04-15T15:16:30.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d27a7e640 1 -- start start 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d27a7e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015e700 0x7f4d200b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d27a7e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d200d2360 0x7f4d200d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d27a7e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d200b47a0 con 0x7f4d2015eae0 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d27a7e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4d200b53b0 con 0x7f4d200d2360 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d27a7e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4d200b4a80 con 0x7f4d2015e700 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d25019640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d200d2360 0x7f4d200d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d2601b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4d2015eae0 0x7f4d200cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32958/0 (socket says 172.21.15.46:32958) 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.371+0000 7f4d2601b640 1 -- 172.21.15.46:0/3732146630 learned_addr learned my addr 172.21.15.46:0/3732146630 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d25019640 1 -- 172.21.15.46:0/3732146630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015e700 msgr2=0x7f4d200b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d2581a640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015e700 0x7f4d200b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d25019640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015e700 0x7f4d200b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d25019640 1 -- 172.21.15.46:0/3732146630 >> v1:172.21.15.46:6789/0 conn(0x7f4d2015eae0 legacy=0x7f4d200cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d25019640 1 -- 172.21.15.46:0/3732146630 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d200b4d60 con 0x7f4d200d2360 2024-04-15T15:16:30.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d24818640 1 -- 172.21.15.46:0/3732146630 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772950627 0 0) 0x7f4d200b47a0 con 0x7f4d2015eae0 2024-04-15T15:16:30.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.372+0000 7f4d2581a640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015e700 0x7f4d200b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d25019640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d200d2b60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d100097a0 tx=0x7f4d1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2212ceb77974913 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d24818640 1 -- 172.21.15.46:0/3732146630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10035020 con 0x7f4d200d2360 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d24818640 1 -- 172.21.15.46:0/3732146630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d10031830 con 0x7f4d200d2360 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d24818640 1 -- 172.21.15.46:0/3732146630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10031c00 con 0x7f4d200d2360 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 -- 172.21.15.46:0/3732146630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 msgr2=0x7f4d200d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d200d2b60 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d100097a0 tx=0x7f4d1002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 -- 172.21.15.46:0/3732146630 shutdown_connections 2024-04-15T15:16:30.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d200d2b60 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/3732146630 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015e700 0x7f4d200b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 -- 172.21.15.46:0/3732146630 >> 172.21.15.46:0/3732146630 conn(0x7f4d2008bf60 msgr2=0x7f4d200d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 -- 172.21.15.46:0/3732146630 shutdown_connections 2024-04-15T15:16:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.373+0000 7f4d27a7e640 1 -- 172.21.15.46:0/3732146630 wait complete. 2024-04-15T15:16:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.374+0000 7f4d27a7e640 1 Processor -- start 2024-04-15T15:16:30.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 -- start start 2024-04-15T15:16:30.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d20153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2015e700 0x7f4d200b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015eae0 0x7f4d200c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4d200b5d80 con 0x7f4d200d2360 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4d200b4450 con 0x7f4d2015e700 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d27a7e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4d200d1d40 con 0x7f4d2015eae0 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d2581a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d20153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d2581a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d20153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43770/0 (socket says 172.21.15.46:43770) 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.375+0000 7f4d2581a640 1 -- 172.21.15.46:0/2757807607 learned_addr learned my addr 172.21.15.46:0/2757807607 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2601b640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015eae0 0x7f4d200c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d25019640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2015e700 0x7f4d200b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2581a640 1 -- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015eae0 msgr2=0x7f4d200c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2581a640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015eae0 0x7f4d200c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2581a640 1 -- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2015e700 msgr2=0x7f4d200b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2581a640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2015e700 0x7f4d200b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2581a640 1 -- 172.21.15.46:0/2757807607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2015f100 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2601b640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015eae0 0x7f4d200c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d25019640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2015e700 0x7f4d200b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d2581a640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d20153cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c7f0 tx=0x7f4d1c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c005b90 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1c005d30 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2015f3e0 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c007a30 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.376+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2015f9a0 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.377+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d200689e0 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.378+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d1c007bd0 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.378+0000 7f4d167fc640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d0406c920 0x7f4d0406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.378+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4d1c08bf60 con 0x7f4d200d2360 2024-04-15T15:16:30.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.378+0000 7f4d25019640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d0406c920 0x7f4d0406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.379+0000 7f4d25019640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d0406c920 0x7f4d0406ede0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4d10036000 tx=0x7f4d10008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.383+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d1c015020 con 0x7f4d200d2360 2024-04-15T15:16:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.567+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4d200cf640 con 0x7f4d200d2360 2024-04-15T15:16:30.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.568+0000 7f4d167fc640 1 -- 172.21.15.46:0/2757807607 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f4d1c05a020 con 0x7f4d200d2360 2024-04-15T15:16:30.570 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d0406c920 msgr2=0x7f4d0406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d0406c920 0x7f4d0406ede0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4d10036000 tx=0x7f4d10008030 comp rx=0 tx=0).stop 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 msgr2=0x7f4d20153cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d20153cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00c7f0 tx=0x7f4d1c00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 shutdown_connections 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4d0406c920 0x7f4d0406ede0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4d2015eae0 0x7f4d200c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d2015e700 0x7f4d200b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 --2- 172.21.15.46:0/2757807607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d200d2360 0x7f4d20153cc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 >> 172.21.15.46:0/2757807607 conn(0x7f4d2008bf60 msgr2=0x7f4d200d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 shutdown_connections 2024-04-15T15:16:30.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.570+0000 7f4d27a7e640 1 -- 172.21.15.46:0/2757807607 wait complete. 2024-04-15T15:16:30.583 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:30.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.691+0000 7f2801ae4640 1 Processor -- start 2024-04-15T15:16:30.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.691+0000 7f2801ae4640 1 -- start start 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f2801ae4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15e700 0x7f27fc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f2801ae4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27fc0d2360 0x7f27fc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f2801ae4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27fc0b47a0 con 0x7f27fc15eae0 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f2801ae4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f27fc0b53b0 con 0x7f27fc0d2360 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f2801ae4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f27fc0b4a80 con 0x7f27fc15e700 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27fc0d2360 0x7f27fc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27fc0d2360 0x7f27fc0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43786/0 (socket says 172.21.15.46:43786) 2024-04-15T15:16:30.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 -- 172.21.15.46:0/2135615152 learned_addr learned my addr 172.21.15.46:0/2135615152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:30.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27fb7fe640 1 --2- 172.21.15.46:0/2135615152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15e700 0x7f27fc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 -- 172.21.15.46:0/2135615152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15e700 msgr2=0x7f27fc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 --2- 172.21.15.46:0/2135615152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15e700 0x7f27fc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 -- 172.21.15.46:0/2135615152 >> v1:172.21.15.78:6789/0 conn(0x7f27fc15eae0 legacy=0x7f27fc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.692+0000 7f27faffd640 1 -- 172.21.15.46:0/2135615152 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27fc0b4d60 con 0x7f27fc0d2360 2024-04-15T15:16:30.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.693+0000 7f27faffd640 1 --2- 172.21.15.46:0/2135615152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc0d2b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009870 tx=0x7f27e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f565be3fffc78c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.693+0000 7f27fa7fc640 1 -- 172.21.15.46:0/2135615152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4034070 con 0x7f27fc0d2360 2024-04-15T15:16:30.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.693+0000 7f27fa7fc640 1 -- 172.21.15.46:0/2135615152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e4030840 con 0x7f27fc0d2360 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.693+0000 7f27fa7fc640 1 -- 172.21.15.46:0/2135615152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4030c10 con 0x7f27fc0d2360 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 -- 172.21.15.46:0/2135615152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 msgr2=0x7f27fc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 --2- 172.21.15.46:0/2135615152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc0d2b60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f27e4009870 tx=0x7f27e402b980 comp rx=0 tx=0).stop 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 -- 172.21.15.46:0/2135615152 shutdown_connections 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 --2- 172.21.15.46:0/2135615152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc0d2b60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 --2- 172.21.15.46:0/2135615152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15e700 0x7f27fc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 -- 172.21.15.46:0/2135615152 >> 172.21.15.46:0/2135615152 conn(0x7f27fc08bf60 msgr2=0x7f27fc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 -- 172.21.15.46:0/2135615152 shutdown_connections 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.694+0000 7f2801ae4640 1 -- 172.21.15.46:0/2135615152 wait complete. 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.695+0000 7f2801ae4640 1 Processor -- start 2024-04-15T15:16:30.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.695+0000 7f2801ae4640 1 -- start start 2024-04-15T15:16:30.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc15e700 0x7f27fc0c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15eae0 0x7f27fc14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc0b5d80 con 0x7f27fc0d2360 2024-04-15T15:16:30.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f27fc0b4450 con 0x7f27fc15e700 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f27fc0d1d40 con 0x7f27fc15eae0 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43790/0 (socket says 172.21.15.46:43790) 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 -- 172.21.15.46:0/3524182561 learned_addr learned my addr 172.21.15.46:0/3524182561 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27faffd640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc15e700 0x7f27fc0c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fbfff640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15eae0 0x7f27fc14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 -- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15eae0 msgr2=0x7f27fc14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15eae0 0x7f27fc14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 -- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc15e700 msgr2=0x7f27fc0c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc15e700 0x7f27fc0c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 -- 172.21.15.46:0/3524182561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27fc0ca2e0 con 0x7f27fc0d2360 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27fb7fe640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc14e550 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00e900 tx=0x7f27ec00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec017070 con 0x7f27fc0d2360 2024-04-15T15:16:30.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ec00ca00 con 0x7f27fc0d2360 2024-04-15T15:16:30.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.696+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27fc0ca5c0 con 0x7f27fc0d2360 2024-04-15T15:16:30.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.697+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00cdb0 con 0x7f27fc0d2360 2024-04-15T15:16:30.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.697+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27fc0cab80 con 0x7f27fc0d2360 2024-04-15T15:16:30.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.697+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27fc0689e0 con 0x7f27fc0d2360 2024-04-15T15:16:30.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.698+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27ec01a070 con 0x7f27fc0d2360 2024-04-15T15:16:30.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.698+0000 7f27f3fff640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f27dc06c8f0 0x7f27dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.699+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f27ec08c340 con 0x7f27fc0d2360 2024-04-15T15:16:30.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.699+0000 7f27faffd640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f27dc06c8f0 0x7f27dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.699+0000 7f27faffd640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f27dc06c8f0 0x7f27dc06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f27e4036000 tx=0x7f27e403b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.703+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27ec05a380 con 0x7f27fc0d2360 2024-04-15T15:16:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.888+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f27fc0ba500 con 0x7f27fc0d2360 2024-04-15T15:16:30.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.889+0000 7f27f3fff640 1 -- 172.21.15.46:0/3524182561 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f27ec059c80 con 0x7f27fc0d2360 2024-04-15T15:16:30.891 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f27dc06c8f0 msgr2=0x7f27dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f27dc06c8f0 0x7f27dc06edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f27e4036000 tx=0x7f27e403b040 comp rx=0 tx=0).stop 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 msgr2=0x7f27fc14e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc14e550 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f27ec00e900 tx=0x7f27ec00edc0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 shutdown_connections 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f27dc06c8f0 0x7f27dc06edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f27fc15eae0 0x7f27fc14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27fc15e700 0x7f27fc0c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 --2- 172.21.15.46:0/3524182561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27fc0d2360 0x7f27fc14e550 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 >> 172.21.15.46:0/3524182561 conn(0x7f27fc08bf60 msgr2=0x7f27fc06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 shutdown_connections 2024-04-15T15:16:30.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:30.891+0000 7f2801ae4640 1 -- 172.21.15.46:0/3524182561 wait complete. 2024-04-15T15:16:30.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:31.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.009+0000 7fa4d8e4a640 1 Processor -- start 2024-04-15T15:16:31.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.009+0000 7fa4d8e4a640 1 -- start start 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.010+0000 7fa4d8e4a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4d415e700 0x7fa4d40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.010+0000 7fa4d8e4a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa4d40d2360 0x7fa4d40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.010+0000 7fa4d8e4a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d40b47a0 con 0x7fa4d415eae0 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.010+0000 7fa4d8e4a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d40b53b0 con 0x7fa4d415e700 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.010+0000 7fa4d8e4a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d40b4a80 con 0x7fa4d40d2360 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4d415e700 0x7fa4d40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa4d415e700 0x7fa4d40b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43796/0 (socket says 172.21.15.46:43796) 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 -- 172.21.15.46:0/3983993668 learned_addr learned my addr 172.21.15.46:0/3983993668 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d1d9b640 1 --2- 172.21.15.46:0/3983993668 >> v2:172.21.15.78:3300/0 conn(0x7fa4d40d2360 0x7fa4d40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 -- 172.21.15.46:0/3983993668 >> v2:172.21.15.78:3300/0 conn(0x7fa4d40d2360 msgr2=0x7fa4d40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 --2- 172.21.15.46:0/3983993668 >> v2:172.21.15.78:3300/0 conn(0x7fa4d40d2360 0x7fa4d40d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 -- 172.21.15.46:0/3983993668 >> v1:172.21.15.78:6789/0 conn(0x7fa4d415eae0 legacy=0x7fa4d40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.011+0000 7fa4d259c640 1 -- 172.21.15.46:0/3983993668 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d40b4d60 con 0x7fa4d415e700 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d259c640 1 --2- 172.21.15.46:0/3983993668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d415e700 0x7fa4d40b5840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa4d408c7b0 tx=0x7fa4cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369571ae405fe33b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d159a640 1 -- 172.21.15.46:0/3983993668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc004230 con 0x7fa4d415e700 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d159a640 1 -- 172.21.15.46:0/3983993668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4cc0043d0 con 0x7fa4d415e700 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d159a640 1 -- 172.21.15.46:0/3983993668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc030840 con 0x7fa4d415e700 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/3983993668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d415e700 msgr2=0x7fa4d40b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/3983993668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d415e700 0x7fa4d40b5840 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa4d408c7b0 tx=0x7fa4cc02b980 comp rx=0 tx=0).stop 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/3983993668 shutdown_connections 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/3983993668 >> v2:172.21.15.78:3300/0 conn(0x7fa4d40d2360 0x7fa4d40d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/3983993668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d415e700 0x7fa4d40b5840 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/3983993668 >> 172.21.15.46:0/3983993668 conn(0x7fa4d408bf60 msgr2=0x7fa4d40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:31.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.012+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/3983993668 shutdown_connections 2024-04-15T15:16:31.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.013+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/3983993668 wait complete. 2024-04-15T15:16:31.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.014+0000 7fa4d8e4a640 1 Processor -- start 2024-04-15T15:16:31.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.014+0000 7fa4d8e4a640 1 -- start start 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.014+0000 7fa4d8e4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 0x7fa4d40b8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.014+0000 7fa4d8e4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d415e700 0x7fa4d4159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d8e4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d415eae0 0x7fa4d40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d8e4a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d40b5d80 con 0x7fa4d40d2360 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d8e4a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d40b4450 con 0x7fa4d415eae0 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d8e4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d40d1d40 con 0x7fa4d415e700 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 0x7fa4d40b8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 0x7fa4d40b8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43800/0 (socket says 172.21.15.46:43800) 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 -- 172.21.15.46:0/2656149377 learned_addr learned my addr 172.21.15.46:0/2656149377 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d2d9d640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d415eae0 0x7fa4d40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d1d9b640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d415e700 0x7fa4d4159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 -- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d415e700 msgr2=0x7fa4d4159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d415e700 0x7fa4d4159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 -- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d415eae0 msgr2=0x7fa4d40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d415eae0 0x7fa4d40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 -- 172.21.15.46:0/2656149377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d415a200 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d259c640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 0x7fa4d40b8be0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc000c00 tx=0x7fa4cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc048070 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4cc030c00 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d40c8350 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc003610 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.015+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d40c8bf0 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.017+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d40c8740 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.017+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4cc04b070 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.017+0000 7fa4caffd640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.018+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa4cc0ae010 con 0x7fa4d40d2360 2024-04-15T15:16:31.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.018+0000 7fa4d1d9b640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.018+0000 7fa4d1d9b640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40d0670 tx=0x7fa4bc009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.022+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4cc034020 con 0x7fa4d40d2360 2024-04-15T15:16:31.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.221+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa4d40689e0 con 0x7fa4d40d2360 2024-04-15T15:16:31.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.221+0000 7fa4caffd640 1 -- 172.21.15.46:0/2656149377 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fa4cc07bba0 con 0x7fa4d40d2360 2024-04-15T15:16:31.223 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 msgr2=0x7fa4b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40d0670 tx=0x7fa4bc009190 comp rx=0 tx=0).stop 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 msgr2=0x7fa4d40b8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 0x7fa4d40b8be0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc000c00 tx=0x7fa4cc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 shutdown_connections 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa4b406c8f0 0x7fa4b406edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4d415eae0 0x7fa4d40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4d415e700 0x7fa4d4159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 --2- 172.21.15.46:0/2656149377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4d40d2360 0x7fa4d40b8be0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 >> 172.21.15.46:0/2656149377 conn(0x7fa4d408bf60 msgr2=0x7fa4d415ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 shutdown_connections 2024-04-15T15:16:31.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.223+0000 7fa4d8e4a640 1 -- 172.21.15.46:0/2656149377 wait complete. 2024-04-15T15:16:31.236 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:16:31.236 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:31.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:31.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.396+0000 7f5499a74640 1 Processor -- start 2024-04-15T15:16:31.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.396+0000 7f5499a74640 1 -- start start 2024-04-15T15:16:31.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f5499a74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f5494070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f5499a74640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549406ff30 con 0x7f5494071d90 2024-04-15T15:16:31.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f5499a74640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54940cecc0 con 0x7f549409b0e0 2024-04-15T15:16:31.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f5499a74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5494070820 con 0x7f54940ce8e0 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f5492ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5494071d90 0x7f549409a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32972/0 (socket says 172.21.15.46:32972) 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f5492ffd640 1 -- 172.21.15.46:0/4054996156 learned_addr learned my addr 172.21.15.46:0/4054996156 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54937fe640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f5494070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4078107786 0 0) 0x7f549406ff30 con 0x7f5494071d90 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f547c003520 con 0x7f5494071d90 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1464351785 0 0) 0x7f547c003520 con 0x7f5494071d90 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 msgr2=0x7f5494070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f5494070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 >> v1:172.21.15.78:6789/0 conn(0x7f549409b0e0 legacy=0x7f54940cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549409b9d0 con 0x7f5494071d90 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f5488001690 con 0x7f5494071d90 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 >> v1:172.21.15.46:6789/0 conn(0x7f5494071d90 legacy=0x7f549409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f547c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c003f60 0x7f547c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f547c004d40 0x7f547c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5494070820 con 0x7f547c004d40 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f547c00bb50 con 0x7f547c003f60 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.397+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f547c00bd50 con 0x7f54940ce8e0 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f54937fe640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f547c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5493fff640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f547c004d40 0x7f547c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5492ffd640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c003f60 0x7f547c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5493fff640 1 -- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 msgr2=0x7f547c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5493fff640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f547c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5493fff640 1 -- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c003f60 msgr2=0x7f547c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5493fff640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c003f60 0x7f547c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f5493fff640 1 -- 172.21.15.46:0/4054996156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c003520 con 0x7f547c004d40 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.398+0000 7f54937fe640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940ce8e0 0x7f547c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:31.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5493fff640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f547c004d40 0x7f547c0051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f548c001e20 tx=0x7f547c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9797193923981e29 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c001a70 con 0x7f547c004d40 2024-04-15T15:16:31.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f548c003b70 con 0x7f547c004d40 2024-04-15T15:16:31.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f54927fc640 1 -- 172.21.15.46:0/4054996156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c0050d0 con 0x7f547c004d40 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5499a74640 1 -- 172.21.15.46:0/4054996156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f547c004d40 msgr2=0x7f547c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5499a74640 1 --2- 172.21.15.46:0/4054996156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f547c004d40 0x7f547c0051c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f548c001e20 tx=0x7f547c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5493fff640 1 -- 172.21.15.46:0/4054996156 reap_dead start 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5499a74640 1 -- 172.21.15.46:0/4054996156 shutdown_connections 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5499a74640 1 -- 172.21.15.46:0/4054996156 >> 172.21.15.46:0/4054996156 conn(0x7f54940b3f40 msgr2=0x7f54940b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5499a74640 1 -- 172.21.15.46:0/4054996156 shutdown_connections 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.399+0000 7f5499a74640 1 -- 172.21.15.46:0/4054996156 wait complete. 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.400+0000 7f5499a74640 1 Processor -- start 2024-04-15T15:16:31.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 -- start start 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c004d40 0x7f54940cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 0x7f5494155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940c7d90 0x7f54940c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54940704b0 con 0x7f54940b7a90 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f54940701d0 con 0x7f547c004d40 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.401+0000 7f5499a74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5494071970 con 0x7f54940c7d90 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 0x7f5494155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 0x7f5494155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43822/0 (socket says 172.21.15.46:43822) 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 -- 172.21.15.46:0/4148036316 learned_addr learned my addr 172.21.15.46:0/4148036316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f54937fe640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c004d40 0x7f54940cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5493fff640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940c7d90 0x7f54940c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 -- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940c7d90 msgr2=0x7f54940c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940c7d90 0x7f54940c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 -- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c004d40 msgr2=0x7f54940cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c004d40 0x7f54940cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 -- 172.21.15.46:0/4148036316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54940cc550 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5493fff640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940c7d90 0x7f54940c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5492ffd640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 0x7f5494155990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5488007290 tx=0x7f5488007690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548800f9c0 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5488018070 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5494155ed0 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548800fda0 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.402+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54940c6a40 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.403+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54940ca340 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.404+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f548801d460 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.404+0000 7f547bfff640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f548006c920 0x7f548006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.404+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f548808bf00 con 0x7f54940b7a90 2024-04-15T15:16:31.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.404+0000 7f54937fe640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f548006c920 0x7f548006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.405+0000 7f54937fe640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f548006c920 0x7f548006ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5484009d60 tx=0x7f5484009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.409+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54940ca340 con 0x7f54940b7a90 2024-04-15T15:16:31.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.594+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54940689e0 con 0x7f54940b7a90 2024-04-15T15:16:31.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.595+0000 7f547bfff640 1 -- 172.21.15.46:0/4148036316 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f5494155ed0 con 0x7f54940b7a90 2024-04-15T15:16:31.597 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:31.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f548006c920 msgr2=0x7f548006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f548006c920 0x7f548006ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5484009d60 tx=0x7f5484009190 comp rx=0 tx=0).stop 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 msgr2=0x7f5494155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 0x7f5494155990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5488007290 tx=0x7f5488007690 comp rx=0 tx=0).stop 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 shutdown_connections 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f548006c920 0x7f548006ede0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54940c7d90 0x7f54940c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54940b7a90 0x7f5494155990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 --2- 172.21.15.46:0/4148036316 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f547c004d40 0x7f54940cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.597+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 >> 172.21.15.46:0/4148036316 conn(0x7f54940b3f40 msgr2=0x7f54941601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.598+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 shutdown_connections 2024-04-15T15:16:31.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.598+0000 7f5499a74640 1 -- 172.21.15.46:0/4148036316 wait complete. 2024-04-15T15:16:31.612 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:31.613 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.719+0000 7fe8a3af8640 1 Processor -- start 2024-04-15T15:16:31.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.719+0000 7fe8a3af8640 1 -- start start 2024-04-15T15:16:31.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a3af8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c15e790 0x7fe89c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a3af8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe89c0b7be0 0x7fe89c0d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a3af8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe89c15eb70 con 0x7fe89c0d1b20 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a3af8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe89c0d3790 con 0x7fe89c0b7be0 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a3af8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe89c0d45b0 con 0x7fe89c15e790 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a1093640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe89c0b7be0 0x7fe89c0d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a2095640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe89c0d1b20 0x7fe89c0d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32978/0 (socket says 172.21.15.46:32978) 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a2095640 1 -- 172.21.15.46:0/3779171890 learned_addr learned my addr 172.21.15.46:0/3779171890 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.720+0000 7fe8a1894640 1 --2- 172.21.15.46:0/3779171890 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c15e790 0x7fe89c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.721+0000 7fe8a1093640 1 -- 172.21.15.46:0/3779171890 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c15e790 msgr2=0x7fe89c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.721+0000 7fe8a1093640 1 --2- 172.21.15.46:0/3779171890 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c15e790 0x7fe89c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.721+0000 7fe8a1093640 1 -- 172.21.15.46:0/3779171890 >> v1:172.21.15.46:6789/0 conn(0x7fe89c0d1b20 legacy=0x7fe89c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.721+0000 7fe8a1093640 1 -- 172.21.15.46:0/3779171890 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c0d3c80 con 0x7fe89c0b7be0 2024-04-15T15:16:31.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.721+0000 7fe8a0892640 1 -- 172.21.15.46:0/3779171890 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3806838139 0 0) 0x7fe89c15eb70 con 0x7fe89c0d1b20 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.721+0000 7fe8a1093640 1 --2- 172.21.15.46:0/3779171890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c0d15e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe8900097a0 tx=0x7fe89002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ab9e061579179d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a0892640 1 -- 172.21.15.46:0/3779171890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890038070 con 0x7fe89c0b7be0 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a0892640 1 -- 172.21.15.46:0/3779171890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe890031700 con 0x7fe89c0b7be0 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a0892640 1 -- 172.21.15.46:0/3779171890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe890031ad0 con 0x7fe89c0b7be0 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 -- 172.21.15.46:0/3779171890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 msgr2=0x7fe89c0d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/3779171890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c0d15e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe8900097a0 tx=0x7fe89002b7a0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 -- 172.21.15.46:0/3779171890 shutdown_connections 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/3779171890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c0d15e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/3779171890 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c15e790 0x7fe89c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 -- 172.21.15.46:0/3779171890 >> 172.21.15.46:0/3779171890 conn(0x7fe89c067b40 msgr2=0x7fe89c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 -- 172.21.15.46:0/3779171890 shutdown_connections 2024-04-15T15:16:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.722+0000 7fe8a3af8640 1 -- 172.21.15.46:0/3779171890 wait complete. 2024-04-15T15:16:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.723+0000 7fe8a3af8640 1 Processor -- start 2024-04-15T15:16:31.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.723+0000 7fe8a3af8640 1 -- start start 2024-04-15T15:16:31.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a3af8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a3af8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c0d1b20 0x7fe89c1f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a3af8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89c15e790 0x7fe89c1f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a3af8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c0d5150 con 0x7fe89c0b7be0 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a3af8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c0d4240 con 0x7fe89c15e790 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a3af8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe89c0d52d0 con 0x7fe89c0d1b20 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a1894640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a1894640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43846/0 (socket says 172.21.15.46:43846) 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a1894640 1 -- 172.21.15.46:0/1087912652 learned_addr learned my addr 172.21.15.46:0/1087912652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a1894640 1 -- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c0d1b20 msgr2=0x7fe89c1f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a2095640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89c15e790 0x7fe89c1f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.724+0000 7fe8a1093640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c0d1b20 0x7fe89c1f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a1894640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c0d1b20 0x7fe89c1f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a1894640 1 -- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89c15e790 msgr2=0x7fe89c1f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a1894640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89c15e790 0x7fe89c1f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a1894640 1 -- 172.21.15.46:0/1087912652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c1f4e60 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a2095640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89c15e790 0x7fe89c1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a1093640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c0d1b20 0x7fe89c1f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a1894640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c159280 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe88c007ef0 tx=0x7fe88c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c014620 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88c019070 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89c1f18b0 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c014a00 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.725+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89c0cc7d0 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.726+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe89c0689e0 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.726+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe88c014ba0 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.727+0000 7fe89a7fc640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe88806c8f0 0x7fe88806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.727+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe88c08c830 con 0x7fe89c0b7be0 2024-04-15T15:16:31.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.727+0000 7fe8a1093640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe88806c8f0 0x7fe88806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:31.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.728+0000 7fe8a1093640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe88806c8f0 0x7fe88806edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe890035000 tx=0x7fe890008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:31.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.732+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe89c1f4e60 con 0x7fe89c0b7be0 2024-04-15T15:16:31.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.917+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe89c0b9660 con 0x7fe89c0b7be0 2024-04-15T15:16:31.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.917+0000 7fe89a7fc640 1 -- 172.21.15.46:0/1087912652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fe88c05a860 con 0x7fe89c0b7be0 2024-04-15T15:16:31.919 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:31.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe88806c8f0 msgr2=0x7fe88806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe88806c8f0 0x7fe88806edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe890035000 tx=0x7fe890008030 comp rx=0 tx=0).stop 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 msgr2=0x7fe89c159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c159280 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe88c007ef0 tx=0x7fe88c00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 shutdown_connections 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe88806c8f0 0x7fe88806edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89c15e790 0x7fe89c1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe89c0d1b20 0x7fe89c1f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 --2- 172.21.15.46:0/1087912652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe89c0b7be0 0x7fe89c159280 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 >> 172.21.15.46:0/1087912652 conn(0x7fe89c067b40 msgr2=0x7fe89c0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:31.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 shutdown_connections 2024-04-15T15:16:31.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:31.919+0000 7fe8a3af8640 1 -- 172.21.15.46:0/1087912652 wait complete. 2024-04-15T15:16:31.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:32.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 Processor -- start 2024-04-15T15:16:32.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 -- start start 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2270070940 0x7f227009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2270160310 0x7f2270070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22700d05e0 con 0x7f22700d0da0 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f2270071260 con 0x7f2270160310 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.043+0000 7f2275340640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f22700d1940 con 0x7f2270070940 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2270160310 0x7f2270070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2270160310 0x7f2270070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43856/0 (socket says 172.21.15.46:43856) 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 -- 172.21.15.46:0/1071897768 learned_addr learned my addr 172.21.15.46:0/1071897768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:32.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 -- 172.21.15.46:0/1071897768 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2270070940 msgr2=0x7f227009b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226effd640 1 --2- 172.21.15.46:0/1071897768 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2270070940 0x7f227009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 --2- 172.21.15.46:0/1071897768 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2270070940 0x7f227009b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 -- 172.21.15.46:0/1071897768 >> v1:172.21.15.78:6789/0 conn(0x7f22700d0da0 legacy=0x7f22700d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.044+0000 7f226e7fc640 1 -- 172.21.15.46:0/1071897768 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227015f720 con 0x7f2270160310 2024-04-15T15:16:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f226e7fc640 1 --2- 172.21.15.46:0/1071897768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f2270070d20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2264009870 tx=0x7f226402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295773afb37aecef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f226dffb640 1 -- 172.21.15.46:0/1071897768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2264035020 con 0x7f2270160310 2024-04-15T15:16:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f226dffb640 1 -- 172.21.15.46:0/1071897768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2264031640 con 0x7f2270160310 2024-04-15T15:16:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f226dffb640 1 -- 172.21.15.46:0/1071897768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226402d040 con 0x7f2270160310 2024-04-15T15:16:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 -- 172.21.15.46:0/1071897768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 msgr2=0x7f2270070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 --2- 172.21.15.46:0/1071897768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f2270070d20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2264009870 tx=0x7f226402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 -- 172.21.15.46:0/1071897768 shutdown_connections 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 --2- 172.21.15.46:0/1071897768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f2270070d20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 --2- 172.21.15.46:0/1071897768 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2270070940 0x7f227009b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 -- 172.21.15.46:0/1071897768 >> 172.21.15.46:0/1071897768 conn(0x7f22700b3f40 msgr2=0x7f22700b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 -- 172.21.15.46:0/1071897768 shutdown_connections 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.045+0000 7f2275340640 1 -- 172.21.15.46:0/1071897768 wait complete. 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 Processor -- start 2024-04-15T15:16:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 -- start start 2024-04-15T15:16:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2270070940 0x7f227015ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f22700d0da0 0x7f227015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f22701f8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f22700d0880 con 0x7f2270160310 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f227015fce0 con 0x7f2270070940 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.047+0000 7f2275340640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2270160770 con 0x7f22700d0da0 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f22701f8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f22701f8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43870/0 (socket says 172.21.15.46:43870) 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 -- 172.21.15.46:0/1656359446 learned_addr learned my addr 172.21.15.46:0/1656359446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226e7fc640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f22700d0da0 0x7f227015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226effd640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2270070940 0x7f227015ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 -- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f22700d0da0 msgr2=0x7f227015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f22700d0da0 0x7f227015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 -- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2270070940 msgr2=0x7f227015ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2270070940 0x7f227015ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 -- 172.21.15.46:0/1656359446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22701f56d0 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226effd640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2270070940 0x7f227015ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226e7fc640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f22700d0da0 0x7f227015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f226f7fe640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f22701f8c80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f226800b970 tx=0x7f226800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268019070 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f226800cb40 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22701f59b0 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268014440 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.048+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22701f8480 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.049+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22700689e0 con 0x7f2270160310 2024-04-15T15:16:32.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.050+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f226801c070 con 0x7f2270160310 2024-04-15T15:16:32.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.050+0000 7f22577fe640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f225c06c8f0 0x7f225c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.050+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f226808bd30 con 0x7f2270160310 2024-04-15T15:16:32.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.051+0000 7f226effd640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f225c06c8f0 0x7f225c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.051+0000 7f226effd640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f225c06c8f0 0x7f225c06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f226000b820 tx=0x7f2260008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.055+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2268059d50 con 0x7f2270160310 2024-04-15T15:16:32.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.238+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22700b1b90 con 0x7f2270160310 2024-04-15T15:16:32.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.239+0000 7f22577fe640 1 -- 172.21.15.46:0/1656359446 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f2268059650 con 0x7f2270160310 2024-04-15T15:16:32.241 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f225c06c8f0 msgr2=0x7f225c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f225c06c8f0 0x7f225c06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f226000b820 tx=0x7f2260008040 comp rx=0 tx=0).stop 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 msgr2=0x7f22701f8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f22701f8c80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f226800b970 tx=0x7f226800be30 comp rx=0 tx=0).stop 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 shutdown_connections 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f225c06c8f0 0x7f225c06edb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270160310 0x7f22701f8c80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f22700d0da0 0x7f227015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 --2- 172.21.15.46:0/1656359446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2270070940 0x7f227015ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 >> 172.21.15.46:0/1656359446 conn(0x7f22700b3f40 msgr2=0x7f227015ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:32.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 shutdown_connections 2024-04-15T15:16:32.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.241+0000 7f2275340640 1 -- 172.21.15.46:0/1656359446 wait complete. 2024-04-15T15:16:32.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:32.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 Processor -- start 2024-04-15T15:16:32.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 -- start start 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe308070940 0x7fe30809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe308070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3080d05e0 con 0x7fe3080d0da0 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe308071260 con 0x7fe308070940 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.360+0000 7fe30f0e4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe3080d1940 con 0x7fe308160310 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30ce80640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe308070940 0x7fe30809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe307fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe308070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30d681640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe3080d0da0 0x7fe3080d1230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:41446/0 (socket says 172.21.15.46:41446) 2024-04-15T15:16:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30d681640 1 -- 172.21.15.46:0/1071187946 learned_addr learned my addr 172.21.15.46:0/1071187946 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30ce80640 1 -- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 msgr2=0x7fe308070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30ce80640 1 --2- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe308070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30ce80640 1 -- 172.21.15.46:0/1071187946 >> v1:172.21.15.78:6789/0 conn(0x7fe3080d0da0 legacy=0x7fe3080d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe30ce80640 1 -- 172.21.15.46:0/1071187946 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30815f720 con 0x7fe308070940 2024-04-15T15:16:32.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.361+0000 7fe307fff640 1 --2- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe308070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.362+0000 7fe30ce80640 1 --2- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe308070940 0x7fe30809b880 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0097a0 tx=0x7fe2fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80604a7de1f61000 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.362+0000 7fe3077fe640 1 -- 172.21.15.46:0/1071187946 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc035020 con 0x7fe308070940 2024-04-15T15:16:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.362+0000 7fe3077fe640 1 -- 172.21.15.46:0/1071187946 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2fc031830 con 0x7fe308070940 2024-04-15T15:16:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.362+0000 7fe3077fe640 1 -- 172.21.15.46:0/1071187946 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc031c00 con 0x7fe308070940 2024-04-15T15:16:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 -- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe308070940 msgr2=0x7fe30809b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe308070940 0x7fe30809b880 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc0097a0 tx=0x7fe2fc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 -- 172.21.15.46:0/1071187946 shutdown_connections 2024-04-15T15:16:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe308070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/1071187946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe308070940 0x7fe30809b880 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 -- 172.21.15.46:0/1071187946 >> 172.21.15.46:0/1071187946 conn(0x7fe3080b3f40 msgr2=0x7fe3080b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 -- 172.21.15.46:0/1071187946 shutdown_connections 2024-04-15T15:16:32.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.363+0000 7fe30f0e4640 1 -- 172.21.15.46:0/1071187946 wait complete. 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.364+0000 7fe30f0e4640 1 Processor -- start 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.364+0000 7fe30f0e4640 1 -- start start 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30f0e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 0x7fe3081f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30f0e4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3080d0da0 0x7fe30815d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30f0e4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe30815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30f0e4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3080d0880 con 0x7fe308070940 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30f0e4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe30815fce0 con 0x7fe3080d0da0 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30f0e4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe308160770 con 0x7fe308160310 2024-04-15T15:16:32.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30ce80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 0x7fe3081f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30ce80640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 0x7fe3081f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43882/0 (socket says 172.21.15.46:43882) 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30ce80640 1 -- 172.21.15.46:0/3344824356 learned_addr learned my addr 172.21.15.46:0/3344824356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe307fff640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3080d0da0 0x7fe30815d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.365+0000 7fe30d681640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe30815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30ce80640 1 -- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 msgr2=0x7fe30815b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30ce80640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe30815b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30ce80640 1 -- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3080d0da0 msgr2=0x7fe30815d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30ce80640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3080d0da0 0x7fe30815d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30ce80640 1 -- 172.21.15.46:0/3344824356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3080b65a0 con 0x7fe308070940 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30ce80640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 0x7fe3081f4720 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc034040 tx=0x7fe2fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc040070 con 0x7fe308070940 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2fc03e070 con 0x7fe308070940 2024-04-15T15:16:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc035040 con 0x7fe308070940 2024-04-15T15:16:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3080b6880 con 0x7fe308070940 2024-04-15T15:16:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.366+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3080b6e40 con 0x7fe308070940 2024-04-15T15:16:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.367+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3080689e0 con 0x7fe308070940 2024-04-15T15:16:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.368+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2fc038020 con 0x7fe308070940 2024-04-15T15:16:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.368+0000 7fe3057fa640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.368+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe2fc0ad720 con 0x7fe308070940 2024-04-15T15:16:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.368+0000 7fe307fff640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.369+0000 7fe307fff640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009560 tx=0x7fe2f4009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.373+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2fc031800 con 0x7fe308070940 2024-04-15T15:16:32.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.558+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe308070720 con 0x7fe308070940 2024-04-15T15:16:32.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.558+0000 7fe3057fa640 1 -- 172.21.15.46:0/3344824356 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fe2fc07b100 con 0x7fe308070940 2024-04-15T15:16:32.560 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.560+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2ec06c8f0 msgr2=0x7fe2ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.560+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009560 tx=0x7fe2f4009190 comp rx=0 tx=0).stop 2024-04-15T15:16:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.560+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 msgr2=0x7fe3081f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.560+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 0x7fe3081f4720 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc034040 tx=0x7fe2fc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 shutdown_connections 2024-04-15T15:16:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe2ec06c8f0 0x7fe2ec06edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe308160310 0x7fe30815b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3080d0da0 0x7fe30815d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 --2- 172.21.15.46:0/3344824356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe308070940 0x7fe3081f4720 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 >> 172.21.15.46:0/3344824356 conn(0x7fe3080b3f40 msgr2=0x7fe3080b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 shutdown_connections 2024-04-15T15:16:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.561+0000 7fe30f0e4640 1 -- 172.21.15.46:0/3344824356 wait complete. 2024-04-15T15:16:32.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:32.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.679+0000 7feaa2d7c640 1 Processor -- start 2024-04-15T15:16:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa2d7c640 1 -- start start 2024-04-15T15:16:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa2d7c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09a9d0 0x7fea9c09adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa2d7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea9c09b780 0x7fea9c0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa2d7c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9c09b2f0 con 0x7fea9c15ebb0 2024-04-15T15:16:32.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa2d7c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fea9c0cd8c0 con 0x7fea9c09b780 2024-04-15T15:16:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa2d7c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c0ced60 con 0x7fea9c09a9d0 2024-04-15T15:16:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.680+0000 7feaa1319640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea9c09b780 0x7fea9c0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa1319640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fea9c09b780 0x7fea9c0b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43886/0 (socket says 172.21.15.46:43886) 2024-04-15T15:16:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa1319640 1 -- 172.21.15.46:0/2145263915 learned_addr learned my addr 172.21.15.46:0/2145263915 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa0b18640 1 --2- 172.21.15.46:0/2145263915 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09a9d0 0x7fea9c09adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa1319640 1 -- 172.21.15.46:0/2145263915 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09a9d0 msgr2=0x7fea9c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa1319640 1 --2- 172.21.15.46:0/2145263915 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09a9d0 0x7fea9c09adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa1319640 1 -- 172.21.15.46:0/2145263915 >> v1:172.21.15.78:6789/0 conn(0x7fea9c15ebb0 legacy=0x7fea9c06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.681+0000 7feaa1319640 1 -- 172.21.15.46:0/2145263915 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0cccb0 con 0x7fea9c09b780 2024-04-15T15:16:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa1319640 1 --2- 172.21.15.46:0/2145263915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c09b780 0x7fea9c0b76f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009e50 tx=0x7fea8c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114dee8b6198aa3a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7fea9b7fe640 1 -- 172.21.15.46:0/2145263915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c02cb00 con 0x7fea9c09b780 2024-04-15T15:16:32.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7fea9b7fe640 1 -- 172.21.15.46:0/2145263915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c02cca0 con 0x7fea9c09b780 2024-04-15T15:16:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7fea9b7fe640 1 -- 172.21.15.46:0/2145263915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c034610 con 0x7fea9c09b780 2024-04-15T15:16:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 -- 172.21.15.46:0/2145263915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c09b780 msgr2=0x7fea9c0b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/2145263915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c09b780 0x7fea9c0b76f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009e50 tx=0x7fea8c02bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 -- 172.21.15.46:0/2145263915 shutdown_connections 2024-04-15T15:16:32.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/2145263915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c09b780 0x7fea9c0b76f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/2145263915 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09a9d0 0x7fea9c09adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 -- 172.21.15.46:0/2145263915 >> 172.21.15.46:0/2145263915 conn(0x7fea9c0b3fb0 msgr2=0x7fea9c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 -- 172.21.15.46:0/2145263915 shutdown_connections 2024-04-15T15:16:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.682+0000 7feaa2d7c640 1 -- 172.21.15.46:0/2145263915 wait complete. 2024-04-15T15:16:32.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 Processor -- start 2024-04-15T15:16:32.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 -- start start 2024-04-15T15:16:32.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea9c09a9d0 0x7fea9c0cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09b780 0x7fea9c0c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 0x7fea9c070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c0cd550 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.684+0000 7feaa2d7c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c0cd270 con 0x7fea9c09a9d0 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa2d7c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fea9c0b5310 con 0x7fea9c09b780 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 0x7fea9c070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 0x7fea9c070940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43900/0 (socket says 172.21.15.46:43900) 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 -- 172.21.15.46:0/757790162 learned_addr learned my addr 172.21.15.46:0/757790162 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7fea9bfff640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09b780 0x7fea9c0c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa0b18640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea9c09a9d0 0x7fea9c0cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 -- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09b780 msgr2=0x7fea9c0c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09b780 0x7fea9c0c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 -- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea9c09a9d0 msgr2=0x7fea9c0cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea9c09a9d0 0x7fea9c0cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 -- 172.21.15.46:0/757790162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c0c8380 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7fea9bfff640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09b780 0x7fea9c0c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:32.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa0b18640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea9c09a9d0 0x7fea9c0cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa1319640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 0x7fea9c070940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009e50 tx=0x7fea8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c043020 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c02ce50 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9c1f83c0 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c009510 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.685+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9c151eb0 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.686+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9c0690f0 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.687+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea8c046020 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.687+0000 7fea997fa640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea7c06c8f0 0x7fea7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.687+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fea8c0ae490 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.687+0000 7feaa0b18640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea7c06c8f0 0x7fea7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.688+0000 7feaa0b18640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea7c06c8f0 0x7fea7c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fea94009ac0 tx=0x7fea94008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:32.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.691+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea8c04d030 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.876+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fea9c067a30 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.876+0000 7fea997fa640 1 -- 172.21.15.46:0/757790162 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fea8c03c350 con 0x7fea9c1f8ea0 2024-04-15T15:16:32.878 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:32.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea7c06c8f0 msgr2=0x7fea7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea7c06c8f0 0x7fea7c06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fea94009ac0 tx=0x7fea94008040 comp rx=0 tx=0).stop 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 msgr2=0x7fea9c070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 0x7fea9c070940 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009e50 tx=0x7fea8c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 shutdown_connections 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fea7c06c8f0 0x7fea7c06edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea9c1f8ea0 0x7fea9c070940 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fea9c09b780 0x7fea9c0c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 --2- 172.21.15.46:0/757790162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fea9c09a9d0 0x7fea9c0cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.878+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 >> 172.21.15.46:0/757790162 conn(0x7fea9c0b3fb0 msgr2=0x7fea9c15ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.879+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 shutdown_connections 2024-04-15T15:16:32.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.879+0000 7feaa2d7c640 1 -- 172.21.15.46:0/757790162 wait complete. 2024-04-15T15:16:32.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.996+0000 7fc0e07ba640 1 Processor -- start 2024-04-15T15:16:32.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.996+0000 7fc0e07ba640 1 -- start start 2024-04-15T15:16:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0e07ba640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc0d815ea40 0x7fc0d80b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0e07ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0d815ee20 0x7fc0d80b6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0e07ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0d80cff70 con 0x7fc0d80b65c0 2024-04-15T15:16:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0e07ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc0d80b69e0 con 0x7fc0d815ee20 2024-04-15T15:16:32.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0e07ba640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc0d80d0250 con 0x7fc0d815ea40 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0d815ee20 0x7fc0d80b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc0d815ee20 0x7fc0d80b6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43918/0 (socket says 172.21.15.46:43918) 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 -- 172.21.15.46:0/4212833064 learned_addr learned my addr 172.21.15.46:0/4212833064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0de556640 1 --2- 172.21.15.46:0/4212833064 >> v2:172.21.15.78:3300/0 conn(0x7fc0d815ea40 0x7fc0d80b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 -- 172.21.15.46:0/4212833064 >> v2:172.21.15.78:3300/0 conn(0x7fc0d815ea40 msgr2=0x7fc0d80b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 --2- 172.21.15.46:0/4212833064 >> v2:172.21.15.78:3300/0 conn(0x7fc0d815ea40 0x7fc0d80b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 -- 172.21.15.46:0/4212833064 >> v1:172.21.15.78:6789/0 conn(0x7fc0d80b65c0 legacy=0x7fc0d80d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.997+0000 7fc0ddd55640 1 -- 172.21.15.46:0/4212833064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d80d0530 con 0x7fc0d815ee20 2024-04-15T15:16:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.998+0000 7fc0ddd55640 1 --2- 172.21.15.46:0/4212833064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d815ee20 0x7fc0d80b6080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc0d808c9c0 tx=0x7fc0c802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1671fa88d0629f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.998+0000 7fc0dd554640 1 -- 172.21.15.46:0/4212833064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c802f850 con 0x7fc0d815ee20 2024-04-15T15:16:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.998+0000 7fc0dd554640 1 -- 172.21.15.46:0/4212833064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0c802f9f0 con 0x7fc0d815ee20 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.998+0000 7fc0dd554640 1 -- 172.21.15.46:0/4212833064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c802fdc0 con 0x7fc0d815ee20 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 -- 172.21.15.46:0/4212833064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d815ee20 msgr2=0x7fc0d80b6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/4212833064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d815ee20 0x7fc0d80b6080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc0d808c9c0 tx=0x7fc0c802ba30 comp rx=0 tx=0).stop 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 -- 172.21.15.46:0/4212833064 shutdown_connections 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/4212833064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d815ee20 0x7fc0d80b6080 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/4212833064 >> v2:172.21.15.78:3300/0 conn(0x7fc0d815ea40 0x7fc0d80b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 -- 172.21.15.46:0/4212833064 >> 172.21.15.46:0/4212833064 conn(0x7fc0d808bf60 msgr2=0x7fc0d80d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 -- 172.21.15.46:0/4212833064 shutdown_connections 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:32.999+0000 7fc0e07ba640 1 -- 172.21.15.46:0/4212833064 wait complete. 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.000+0000 7fc0e07ba640 1 Processor -- start 2024-04-15T15:16:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.000+0000 7fc0e07ba640 1 -- start start 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 0x7fc0d80ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc0d815ea40 0x7fc0d8153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0d815ee20 0x7fc0d80d0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d80b38c0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d80d0af0 con 0x7fc0d815ee20 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc0d80b3a40 con 0x7fc0d815ea40 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 0x7fc0d80ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 0x7fc0d80ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43920/0 (socket says 172.21.15.46:43920) 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 -- 172.21.15.46:0/3568362975 learned_addr learned my addr 172.21.15.46:0/3568362975 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0ddd55640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc0d815ea40 0x7fc0d8153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0ded57640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0d815ee20 0x7fc0d80d0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 -- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc0d815ea40 msgr2=0x7fc0d8153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc0d815ea40 0x7fc0d8153b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 -- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0d815ee20 msgr2=0x7fc0d80d0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0d815ee20 0x7fc0d80d0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 -- 172.21.15.46:0/3568362975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d81f8ca0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0de556640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 0x7fc0d80ce0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400d800 tx=0x7fc0d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400b6c0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4019070 con 0x7fc0d80b65c0 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d81f8090 con 0x7fc0d80b65c0 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400ba80 con 0x7fc0d80b65c0 2024-04-15T15:16:33.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.001+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d81f8650 con 0x7fc0d80b65c0 2024-04-15T15:16:33.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.002+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0d80cfcb0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.003+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0d400bc20 con 0x7fc0d80b65c0 2024-04-15T15:16:33.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.003+0000 7fc0ceffd640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.003+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc0d408c230 con 0x7fc0d80b65c0 2024-04-15T15:16:33.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.004+0000 7fc0ddd55640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.004+0000 7fc0ddd55640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8036000 tx=0x7fc0c8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.008+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0d80cfcb0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.205+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc0d80689e0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.206+0000 7fc0ceffd640 1 -- 172.21.15.46:0/3568362975 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fc0d4059ca0 con 0x7fc0d80b65c0 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc0bc06c8f0 msgr2=0x7fc0bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8036000 tx=0x7fc0c8008030 comp rx=0 tx=0).stop 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 msgr2=0x7fc0d80ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 0x7fc0d80ce0e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc0d400d800 tx=0x7fc0d400dcc0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 shutdown_connections 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc0bc06c8f0 0x7fc0bc06edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0d815ee20 0x7fc0d80d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc0d815ea40 0x7fc0d8153b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 --2- 172.21.15.46:0/3568362975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d80b65c0 0x7fc0d80ce0e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 >> 172.21.15.46:0/3568362975 conn(0x7fc0d808bf60 msgr2=0x7fc0d806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 shutdown_connections 2024-04-15T15:16:33.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.208+0000 7fc0e07ba640 1 -- 172.21.15.46:0/3568362975 wait complete. 2024-04-15T15:16:33.250 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-04-15T15:16:33.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:33.250 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:33.282 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:33.283 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:16:33.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:33.283 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:16:33.339 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:16:33.339 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:33.360 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:16:33.360 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:33.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.522+0000 7f4caba93640 1 Processor -- start 2024-04-15T15:16:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.522+0000 7f4caba93640 1 -- start start 2024-04-15T15:16:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caba93640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ca40d2360 0x7f4ca40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caba93640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca415eae0 0x7f4ca40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caba93640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca40b47a0 con 0x7f4ca415e700 2024-04-15T15:16:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caba93640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca40b53b0 con 0x7f4ca40d2360 2024-04-15T15:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caba93640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca40b4a80 con 0x7f4ca415eae0 2024-04-15T15:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ca40d2360 0x7f4ca40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caa030640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca415eae0 0x7f4ca40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4ca40d2360 0x7f4ca40d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:36642/0 (socket says 172.21.15.46:36642) 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca982f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4ca415e700 0x7f4ca40b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:41470/0 (socket says 172.21.15.46:41470) 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 -- 172.21.15.46:0/3923339184 learned_addr learned my addr 172.21.15.46:0/3923339184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 -- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca415eae0 msgr2=0x7f4ca40d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 --2- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca415eae0 0x7f4ca40d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 -- 172.21.15.46:0/3923339184 >> v1:172.21.15.78:6789/0 conn(0x7f4ca415e700 legacy=0x7f4ca40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4ca902e640 1 -- 172.21.15.46:0/3923339184 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca40b4d60 con 0x7f4ca40d2360 2024-04-15T15:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.523+0000 7f4caa030640 1 --2- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca415eae0 0x7f4ca40d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.524+0000 7f4ca902e640 1 --2- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40d2360 0x7f4ca40d2b60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009870 tx=0x7f4c9402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40c20cb504999458 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.524+0000 7f4ca882d640 1 -- 172.21.15.46:0/3923339184 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94034070 con 0x7f4ca40d2360 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.524+0000 7f4ca882d640 1 -- 172.21.15.46:0/3923339184 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c94030840 con 0x7f4ca40d2360 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.524+0000 7f4ca882d640 1 -- 172.21.15.46:0/3923339184 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94030c10 con 0x7f4ca40d2360 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 -- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40d2360 msgr2=0x7f4ca40d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 --2- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40d2360 0x7f4ca40d2b60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c94009870 tx=0x7f4c9402b980 comp rx=0 tx=0).stop 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 -- 172.21.15.46:0/3923339184 shutdown_connections 2024-04-15T15:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 --2- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca415eae0 0x7f4ca40d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 --2- 172.21.15.46:0/3923339184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca40d2360 0x7f4ca40d2b60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 -- 172.21.15.46:0/3923339184 >> 172.21.15.46:0/3923339184 conn(0x7f4ca408bf60 msgr2=0x7f4ca40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 -- 172.21.15.46:0/3923339184 shutdown_connections 2024-04-15T15:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.525+0000 7f4caba93640 1 -- 172.21.15.46:0/3923339184 wait complete. 2024-04-15T15:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.526+0000 7f4caba93640 1 Processor -- start 2024-04-15T15:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.526+0000 7f4caba93640 1 -- start start 2024-04-15T15:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caba93640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca40d2360 0x7f4ca40cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caba93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 0x7f4ca40caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caba93640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca415eae0 0x7f4ca40b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caba93640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca40d1d40 con 0x7f4ca415e700 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caba93640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca40b4450 con 0x7f4ca415eae0 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caba93640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca40d1ec0 con 0x7f4ca40d2360 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 0x7f4ca40caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 0x7f4ca40caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43926/0 (socket says 172.21.15.46:43926) 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 -- 172.21.15.46:0/2446960744 learned_addr learned my addr 172.21.15.46:0/2446960744 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca982f640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca40d2360 0x7f4ca40cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4caa030640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca415eae0 0x7f4ca40b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 -- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca40d2360 msgr2=0x7f4ca40cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca40d2360 0x7f4ca40cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 -- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca415eae0 msgr2=0x7f4ca40b9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca415eae0 0x7f4ca40b9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.527+0000 7f4ca902e640 1 -- 172.21.15.46:0/2446960744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca40ccd10 con 0x7f4ca415e700 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.528+0000 7f4ca902e640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 0x7f4ca40caa90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4c9402b9c0 tx=0x7f4c94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.528+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94048070 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.528+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c94030640 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.528+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca40ccff0 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.528+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c94034040 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.528+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca40cd5b0 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.529+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca40689e0 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.529+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c9404b070 con 0x7f4ca415e700 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.530+0000 7f4c9a7fc640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c8806c8f0 0x7f4c8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.530+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4c940add00 con 0x7f4ca415e700 2024-04-15T15:16:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.530+0000 7f4ca982f640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c8806c8f0 0x7f4c8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.530+0000 7f4ca982f640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c8806c8f0 0x7f4c8806edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0000fb0 tx=0x7f4ca0007e70 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.534+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c9407b770 con 0x7f4ca415e700 2024-04-15T15:16:33.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.723+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ca40ba500 con 0x7f4ca415e700 2024-04-15T15:16:33.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.724+0000 7f4c9a7fc640 1 -- 172.21.15.46:0/2446960744 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f4c9407fa70 con 0x7f4ca415e700 2024-04-15T15:16:33.726 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:33.726 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c8806c8f0 msgr2=0x7f4c8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c8806c8f0 0x7f4c8806edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0000fb0 tx=0x7f4ca0007e70 comp rx=0 tx=0).stop 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 msgr2=0x7f4ca40caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 0x7f4ca40caa90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4c9402b9c0 tx=0x7f4c94004730 comp rx=0 tx=0).stop 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 shutdown_connections 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c8806c8f0 0x7f4c8806edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ca415eae0 0x7f4ca40b9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ca415e700 0x7f4ca40caa90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 --2- 172.21.15.46:0/2446960744 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4ca40d2360 0x7f4ca40cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 >> 172.21.15.46:0/2446960744 conn(0x7f4ca408bf60 msgr2=0x7f4ca40b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 shutdown_connections 2024-04-15T15:16:33.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.726+0000 7f4caba93640 1 -- 172.21.15.46:0/2446960744 wait complete. 2024-04-15T15:16:33.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:33.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.845+0000 7f9b84a50640 1 Processor -- start 2024-04-15T15:16:33.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.845+0000 7f9b84a50640 1 -- start start 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.846+0000 7f9b84a50640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b8015ebb0 0x7f9b8006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.846+0000 7f9b84a50640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b800ced60 con 0x7f9b8009b780 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.846+0000 7f9b84a50640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b800cccb0 con 0x7f9b8009a9d0 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.846+0000 7f9b84a50640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9b800cd8c0 con 0x7f9b8015ebb0 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.846+0000 7f9b7ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9b8009b780 0x7f9b800b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:32986/0 (socket says 172.21.15.46:32986) 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.846+0000 7f9b7ed9d640 1 -- 172.21.15.46:0/2624735795 learned_addr learned my addr 172.21.15.46:0/2624735795 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7dd9b640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b8015ebb0 0x7f9b8006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 862644446 0 0) 0x7f9b78001110 con 0x7f9b8009b780 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b6c003520 con 0x7f9b8009b780 2024-04-15T15:16:33.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4146686164 0 0) 0x7f9b6c003520 con 0x7f9b8009b780 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b8015ebb0 msgr2=0x7f9b8006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b8015ebb0 0x7f9b8006cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 >> v1:172.21.15.78:6789/0 conn(0x7f9b8009a9d0 legacy=0x7f9b8009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b800ccf90 con 0x7f9b8009b780 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f9b78001840 con 0x7f9b8009b780 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 >> v1:172.21.15.46:6789/0 conn(0x7f9b8009b780 legacy=0x7f9b800b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b6c0039e0 0x7f9b6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c004390 0x7f9b6c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b6c004f80 0x7f9b6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800cd8c0 con 0x7f9b6c0039e0 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c00bf00 con 0x7f9b6c004f80 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c00c080 con 0x7f9b6c004390 2024-04-15T15:16:33.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b6c0039e0 0x7f9b6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7dd9b640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c004390 0x7f9b6c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7ed9d640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b6c004f80 0x7f9b6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7e59c640 1 -- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c004390 msgr2=0x7f9b6c004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c004390 0x7f9b6c004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7e59c640 1 -- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b6c004f80 msgr2=0x7f9b6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.847+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b6c004f80 0x7f9b6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.848+0000 7f9b7e59c640 1 -- 172.21.15.46:0/2624735795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c003520 con 0x7f9b6c0039e0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.848+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b6c0039e0 0x7f9b6c003dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9b70001f60 tx=0x7f9b6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0566bb672473c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.848+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70039070 con 0x7f9b6c0039e0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.848+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7002c730 con 0x7f9b6c0039e0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.848+0000 7f9b7d59a640 1 -- 172.21.15.46:0/2624735795 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70001b30 con 0x7f9b6c0039e0 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b84a50640 1 -- 172.21.15.46:0/2624735795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b6c0039e0 msgr2=0x7f9b6c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b84a50640 1 --2- 172.21.15.46:0/2624735795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b6c0039e0 0x7f9b6c003dc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9b70001f60 tx=0x7f9b6c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b7ed9d640 1 -- 172.21.15.46:0/2624735795 reap_dead start 2024-04-15T15:16:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b84a50640 1 -- 172.21.15.46:0/2624735795 shutdown_connections 2024-04-15T15:16:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b84a50640 1 -- 172.21.15.46:0/2624735795 >> 172.21.15.46:0/2624735795 conn(0x7f9b800b3fb0 msgr2=0x7f9b800b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b84a50640 1 -- 172.21.15.46:0/2624735795 shutdown_connections 2024-04-15T15:16:33.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.849+0000 7f9b84a50640 1 -- 172.21.15.46:0/2624735795 wait complete. 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.850+0000 7f9b84a50640 1 Processor -- start 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.850+0000 7f9b84a50640 1 -- start start 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b84a50640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c0039e0 0x7f9b800b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b84a50640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b801f7470 0x7f9b800c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b84a50640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 0x7f9b800d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b84a50640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800b5310 con 0x7f9b800c9590 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b84a50640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800cd550 con 0x7f9b801f7470 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b84a50640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9b800b5490 con 0x7f9b6c0039e0 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b7ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 0x7f9b800d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b7ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 0x7f9b800d07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43936/0 (socket says 172.21.15.46:43936) 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.851+0000 7f9b7ed9d640 1 -- 172.21.15.46:0/3321300454 learned_addr learned my addr 172.21.15.46:0/3321300454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:33.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c0039e0 0x7f9b800b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7ed9d640 1 -- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c0039e0 msgr2=0x7f9b800b7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7ed9d640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c0039e0 0x7f9b800b7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7ed9d640 1 -- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b801f7470 msgr2=0x7f9b800c8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7dd9b640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b801f7470 0x7f9b800c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7ed9d640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b801f7470 0x7f9b800c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7ed9d640 1 -- 172.21.15.46:0/3321300454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b801f3e80 con 0x7f9b800c9590 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b7ed9d640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 0x7f9b800d07b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9b80162400 tx=0x7f9b78007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7800f910 con 0x7f9b800c9590 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b78018070 con 0x7f9b800c9590 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7800fd30 con 0x7f9b800c9590 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801f7900 con 0x7f9b800c9590 2024-04-15T15:16:33.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.852+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801f7ec0 con 0x7f9b800c9590 2024-04-15T15:16:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.853+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b801f4050 con 0x7f9b800c9590 2024-04-15T15:16:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.854+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b7801c410 con 0x7f9b800c9590 2024-04-15T15:16:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.854+0000 7f9b76ffd640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9b6006c920 0x7f9b6006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.854+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9b7808bbc0 con 0x7f9b800c9590 2024-04-15T15:16:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.854+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9b6006c920 0x7f9b6006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.855+0000 7f9b7e59c640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9b6006c920 0x7f9b6006ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b70036000 tx=0x7f9b70009030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:33.859+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b800ced60 con 0x7f9b800c9590 2024-04-15T15:16:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.043+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9b800690f0 con 0x7f9b800c9590 2024-04-15T15:16:34.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.043+0000 7f9b76ffd640 1 -- 172.21.15.46:0/3321300454 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f9b78059be0 con 0x7f9b800c9590 2024-04-15T15:16:34.045 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9b6006c920 msgr2=0x7f9b6006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9b6006c920 0x7f9b6006ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b70036000 tx=0x7f9b70009030 comp rx=0 tx=0).stop 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 msgr2=0x7f9b800d07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 0x7f9b800d07b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9b80162400 tx=0x7f9b78007720 comp rx=0 tx=0).stop 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 shutdown_connections 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9b6006c920 0x7f9b6006ede0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b800c9590 0x7f9b800d07b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b801f7470 0x7f9b800c8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 --2- 172.21.15.46:0/3321300454 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9b6c0039e0 0x7f9b800b7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 >> 172.21.15.46:0/3321300454 conn(0x7f9b800b3fb0 msgr2=0x7f9b8015cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 shutdown_connections 2024-04-15T15:16:34.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.045+0000 7f9b84a50640 1 -- 172.21.15.46:0/3321300454 wait complete. 2024-04-15T15:16:34.058 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:34.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.164+0000 7fda25ace640 1 Processor -- start 2024-04-15T15:16:34.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.164+0000 7fda25ace640 1 -- start start 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda25ace640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda2006e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda25ace640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda200ccb20 0x7fda2015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda25ace640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda2006eaf0 con 0x7fda2015cf80 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda25ace640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fda2006edd0 con 0x7fda200ccb20 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda25ace640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fda2006f0b0 con 0x7fda2015c660 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1effd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda200ccb20 0x7fda2015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fda2015cf80 0x7fda200cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33002/0 (socket says 172.21.15.46:33002) 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1ffff640 1 -- 172.21.15.46:0/3132166997 learned_addr learned my addr 172.21.15.46:0/3132166997 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1f7fe640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda2006e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1effd640 1 -- 172.21.15.46:0/3132166997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 msgr2=0x7fda2006e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1effd640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda2006e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1effd640 1 -- 172.21.15.46:0/3132166997 >> v1:172.21.15.46:6789/0 conn(0x7fda2015cf80 legacy=0x7fda200cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1effd640 1 -- 172.21.15.46:0/3132166997 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2006f390 con 0x7fda200ccb20 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1e7fc640 1 -- 172.21.15.46:0/3132166997 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 635988035 0 0) 0x7fda2006eaf0 con 0x7fda2015cf80 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.165+0000 7fda1f7fe640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda2006e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.166+0000 7fda1effd640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda200ccb20 0x7fda2015ca40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fda140097a0 tx=0x7fda1402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6668de5c441b0be2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.166+0000 7fda1e7fc640 1 -- 172.21.15.46:0/3132166997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda140315a0 con 0x7fda200ccb20 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.166+0000 7fda1e7fc640 1 -- 172.21.15.46:0/3132166997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda14037070 con 0x7fda200ccb20 2024-04-15T15:16:34.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.166+0000 7fda1e7fc640 1 -- 172.21.15.46:0/3132166997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14031960 con 0x7fda200ccb20 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 -- 172.21.15.46:0/3132166997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda200ccb20 msgr2=0x7fda2015ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda200ccb20 0x7fda2015ca40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fda140097a0 tx=0x7fda1402b840 comp rx=0 tx=0).stop 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 -- 172.21.15.46:0/3132166997 shutdown_connections 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda200ccb20 0x7fda2015ca40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 --2- 172.21.15.46:0/3132166997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda2006e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 -- 172.21.15.46:0/3132166997 >> 172.21.15.46:0/3132166997 conn(0x7fda2006b9e0 msgr2=0x7fda2006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 -- 172.21.15.46:0/3132166997 shutdown_connections 2024-04-15T15:16:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.167+0000 7fda25ace640 1 -- 172.21.15.46:0/3132166997 wait complete. 2024-04-15T15:16:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.168+0000 7fda25ace640 1 Processor -- start 2024-04-15T15:16:34.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.168+0000 7fda25ace640 1 -- start start 2024-04-15T15:16:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda25ace640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda200ccb20 0x7fda20146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda25ace640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda201f1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda25ace640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 0x7fda2014f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda25ace640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda200d0180 con 0x7fda2015cf80 2024-04-15T15:16:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda25ace640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fda200d1110 con 0x7fda200ccb20 2024-04-15T15:16:34.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda25ace640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fda200d0300 con 0x7fda2015c660 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 0x7fda2014f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 0x7fda2014f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43956/0 (socket says 172.21.15.46:43956) 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 -- 172.21.15.46:0/3490039160 learned_addr learned my addr 172.21.15.46:0/3490039160 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 -- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 msgr2=0x7fda201f1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda201f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 -- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda200ccb20 msgr2=0x7fda20146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda200ccb20 0x7fda20146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 -- 172.21.15.46:0/3490039160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda20158550 con 0x7fda2015cf80 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.169+0000 7fda1ffff640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 0x7fda2014f640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c7d0 tx=0x7fda1800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.170+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda18018070 con 0x7fda2015cf80 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.170+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1800a640 con 0x7fda2015cf80 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.170+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1800aa70 con 0x7fda2015cf80 2024-04-15T15:16:34.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.170+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda201587d0 con 0x7fda2015cf80 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.170+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda20158d50 con 0x7fda2015cf80 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.171+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2006c970 con 0x7fda2015cf80 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.171+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda1801b070 con 0x7fda2015cf80 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.171+0000 7fda07fff640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fda0c06c8f0 0x7fda0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.172+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fda1808b470 con 0x7fda2015cf80 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.172+0000 7fda1f7fe640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fda0c06c8f0 0x7fda0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.172+0000 7fda1f7fe640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fda0c06c8f0 0x7fda0c06edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fda10006fd0 tx=0x7fda10008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.176+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda20158550 con 0x7fda2015cf80 2024-04-15T15:16:34.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.359+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fda20146fb0 con 0x7fda2015cf80 2024-04-15T15:16:34.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.360+0000 7fda07fff640 1 -- 172.21.15.46:0/3490039160 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fda18059530 con 0x7fda2015cf80 2024-04-15T15:16:34.362 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fda0c06c8f0 msgr2=0x7fda0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fda0c06c8f0 0x7fda0c06edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fda10006fd0 tx=0x7fda10008040 comp rx=0 tx=0).stop 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 msgr2=0x7fda2014f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 0x7fda2014f640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c7d0 tx=0x7fda1800cc90 comp rx=0 tx=0).stop 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 shutdown_connections 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fda0c06c8f0 0x7fda0c06edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda2015cf80 0x7fda2014f640 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fda2015c660 0x7fda201f1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 --2- 172.21.15.46:0/3490039160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fda200ccb20 0x7fda20146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 >> 172.21.15.46:0/3490039160 conn(0x7fda2006b9e0 msgr2=0x7fda20072080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 shutdown_connections 2024-04-15T15:16:34.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.362+0000 7fda25ace640 1 -- 172.21.15.46:0/3490039160 wait complete. 2024-04-15T15:16:34.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.481+0000 7f7de9688640 1 Processor -- start 2024-04-15T15:16:34.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.481+0000 7f7de9688640 1 -- start start 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de9688640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4158770 0x7f7de415bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de9688640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7de40b5230 0x7f7de4072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de9688640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7de4072c60 0x7f7de415e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de9688640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7de40b4da0 con 0x7f7de4072c60 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de9688640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f7de415c1d0 con 0x7f7de40b5230 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de9688640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7de40b4190 con 0x7f7de4158770 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7de4072c60 0x7f7de415e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7de4072c60 0x7f7de415e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43968/0 (socket says 172.21.15.46:43968) 2024-04-15T15:16:34.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 -- 172.21.15.46:0/3343892973 learned_addr learned my addr 172.21.15.46:0/3343892973 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/3343892973 >> v2:172.21.15.78:3300/0 conn(0x7f7de40b5230 0x7f7de4072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4158770 0x7f7de415bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 -- 172.21.15.46:0/3343892973 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4158770 msgr2=0x7f7de415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4158770 0x7f7de415bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 -- 172.21.15.46:0/3343892973 >> v2:172.21.15.78:3300/0 conn(0x7f7de40b5230 msgr2=0x7f7de4072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3343892973 >> v2:172.21.15.78:3300/0 conn(0x7f7de40b5230 0x7f7de4072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de37fe640 1 -- 172.21.15.46:0/3343892973 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de40b4470 con 0x7f7de4072c60 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.482+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4158770 0x7f7de415bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.483+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4072c60 0x7f7de415e800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7de406feb0 tx=0x7f7ddc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22e5b1b9a081bff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.483+0000 7f7de27fc640 1 -- 172.21.15.46:0/3343892973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc02ca70 con 0x7f7de4072c60 2024-04-15T15:16:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.483+0000 7f7de27fc640 1 -- 172.21.15.46:0/3343892973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc02cc10 con 0x7f7de4072c60 2024-04-15T15:16:34.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.483+0000 7f7de27fc640 1 -- 172.21.15.46:0/3343892973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0343f0 con 0x7f7de4072c60 2024-04-15T15:16:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 -- 172.21.15.46:0/3343892973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4072c60 msgr2=0x7f7de415e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4072c60 0x7f7de415e800 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7de406feb0 tx=0x7f7ddc02bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 -- 172.21.15.46:0/3343892973 shutdown_connections 2024-04-15T15:16:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4072c60 0x7f7de415e800 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 --2- 172.21.15.46:0/3343892973 >> v2:172.21.15.78:3300/0 conn(0x7f7de40b5230 0x7f7de4072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 --2- 172.21.15.46:0/3343892973 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4158770 0x7f7de415bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 -- 172.21.15.46:0/3343892973 >> 172.21.15.46:0/3343892973 conn(0x7f7de406be00 msgr2=0x7f7de406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 -- 172.21.15.46:0/3343892973 shutdown_connections 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.484+0000 7f7de9688640 1 -- 172.21.15.46:0/3343892973 wait complete. 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.485+0000 7f7de9688640 1 Processor -- start 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.485+0000 7f7de9688640 1 -- start start 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de9688640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4072c60 0x7f7de40ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de9688640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7de40b5230 0x7f7de4156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de9688640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 0x7f7de41e7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de9688640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7de4071fa0 con 0x7f7de4158770 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de9688640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7de40b4a30 con 0x7f7de40b5230 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de9688640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7de4158b50 con 0x7f7de4072c60 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 0x7f7de41e7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 0x7f7de41e7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43982/0 (socket says 172.21.15.46:43982) 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 -- 172.21.15.46:0/3083070987 learned_addr learned my addr 172.21.15.46:0/3083070987 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7de40b5230 0x7f7de4156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4072c60 0x7f7de40ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 -- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4072c60 msgr2=0x7f7de40ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4072c60 0x7f7de40ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 -- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7de40b5230 msgr2=0x7f7de4156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7de40b5230 0x7f7de4156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 -- 172.21.15.46:0/3083070987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de409c880 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7dda7fc640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7de40b5230 0x7f7de4156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4072c60 0x7f7de40ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.486+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 0x7f7de41e7c70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc036040 tx=0x7f7ddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.487+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc043020 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.487+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc03e450 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.487+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de409cb60 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.487+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc009510 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.487+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de409ce40 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.488+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de40689e0 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.488+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ddc046020 con 0x7f7de4158770 2024-04-15T15:16:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.489+0000 7f7ddbfff640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7dc406c920 0x7f7dc406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.489+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7ddc0ae6f0 con 0x7f7de4158770 2024-04-15T15:16:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.489+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7dc406c920 0x7f7dc406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.489+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7dc406c920 0x7f7dc406ede0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4006fd0 tx=0x7f7dd4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.493+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ddc04d030 con 0x7f7de4158770 2024-04-15T15:16:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.680+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7de40afb70 con 0x7f7de4158770 2024-04-15T15:16:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.681+0000 7f7ddbfff640 1 -- 172.21.15.46:0/3083070987 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f7ddc07c7b0 con 0x7f7de4158770 2024-04-15T15:16:34.683 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7dc406c920 msgr2=0x7f7dc406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7dc406c920 0x7f7dc406ede0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4006fd0 tx=0x7f7dd4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 msgr2=0x7f7de41e7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 0x7f7de41e7c70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc036040 tx=0x7f7ddc004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 shutdown_connections 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7dc406c920 0x7f7dc406ede0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4158770 0x7f7de41e7c70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7de40b5230 0x7f7de4156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 --2- 172.21.15.46:0/3083070987 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7de4072c60 0x7f7de40ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 >> 172.21.15.46:0/3083070987 conn(0x7f7de406be00 msgr2=0x7f7de406e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 shutdown_connections 2024-04-15T15:16:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.683+0000 7f7de9688640 1 -- 172.21.15.46:0/3083070987 wait complete. 2024-04-15T15:16:34.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:34.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.803+0000 7f1dc984a640 1 Processor -- start 2024-04-15T15:16:34.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.803+0000 7f1dc984a640 1 -- start start 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc984a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1dc415e790 0x7f1dc40d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc984a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1dc40d1610 0x7f1dc40d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc984a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dc40d2480 con 0x7f1dc40cfd20 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc984a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1dc40d2db0 con 0x7f1dc40d1610 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc984a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f1dc40d2760 con 0x7f1dc415e790 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dba7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1dc40cfd20 0x7f1dc40d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33004/0 (socket says 172.21.15.46:33004) 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dba7fc640 1 -- 172.21.15.46:0/3027608412 learned_addr learned my addr 172.21.15.46:0/3027608412 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:34.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/3027608412 >> v2:172.21.15.46:3300/0 conn(0x7f1dc40d1610 0x7f1dc40d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/3027608412 >> v2:172.21.15.78:3300/0 conn(0x7f1dc415e790 0x7f1dc40d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3027608412 >> v2:172.21.15.78:3300/0 conn(0x7f1dc415e790 msgr2=0x7f1dc40d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/3027608412 >> v2:172.21.15.78:3300/0 conn(0x7f1dc415e790 0x7f1dc40d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3027608412 >> v1:172.21.15.46:6789/0 conn(0x7f1dc40cfd20 legacy=0x7f1dc40d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.804+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3027608412 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc40d2a40 con 0x7f1dc40d1610 2024-04-15T15:16:34.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.805+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/3027608412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d1610 0x7f1dc40d0160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009d90 tx=0x7f1dbc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1003c1048e3ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.805+0000 7f1dc27fc640 1 -- 172.21.15.46:0/3027608412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc039070 con 0x7f1dc40d1610 2024-04-15T15:16:34.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.805+0000 7f1dc27fc640 1 -- 172.21.15.46:0/3027608412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dbc02cc60 con 0x7f1dc40d1610 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.805+0000 7f1dc27fc640 1 -- 172.21.15.46:0/3027608412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc034660 con 0x7f1dc40d1610 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 -- 172.21.15.46:0/3027608412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d1610 msgr2=0x7f1dc40d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3027608412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d1610 0x7f1dc40d0160 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009d90 tx=0x7f1dbc02beb0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 -- 172.21.15.46:0/3027608412 shutdown_connections 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3027608412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc40d1610 0x7f1dc40d0160 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3027608412 >> v2:172.21.15.78:3300/0 conn(0x7f1dc415e790 0x7f1dc40d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 -- 172.21.15.46:0/3027608412 >> 172.21.15.46:0/3027608412 conn(0x7f1dc408bf60 msgr2=0x7f1dc40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 -- 172.21.15.46:0/3027608412 shutdown_connections 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.806+0000 7f1dc984a640 1 -- 172.21.15.46:0/3027608412 wait complete. 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.807+0000 7f1dc984a640 1 Processor -- start 2024-04-15T15:16:34.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.807+0000 7f1dc984a640 1 -- start start 2024-04-15T15:16:34.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dc40cfd20 0x7f1dc409e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1dc40d1610 0x7f1dc40b12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 0x7f1dc40b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc415ecf0 con 0x7f1dc415e790 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc415eb70 con 0x7f1dc40cfd20 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1dc40b74f0 con 0x7f1dc40d1610 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 0x7f1dc40b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 0x7f1dc40b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44002/0 (socket says 172.21.15.46:44002) 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3355204918 learned_addr learned my addr 172.21.15.46:0/3355204918 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dc40cfd20 0x7f1dc409e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dba7fc640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1dc40d1610 0x7f1dc40b12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1dc40d1610 msgr2=0x7f1dc40b12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1dc40d1610 0x7f1dc40b12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dc40cfd20 msgr2=0x7f1dc409e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dc40cfd20 0x7f1dc409e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 -- 172.21.15.46:0/3355204918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc40b1830 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dc40cfd20 0x7f1dc409e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc37fe640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 0x7f1dc40b4b20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009d90 tx=0x7f1dbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc039050 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dbc045070 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc40b1b00 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc03e830 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.808+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc40b20c0 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.809+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc40689e0 con 0x7f1dc415e790 2024-04-15T15:16:34.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.810+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1dbc03e9d0 con 0x7f1dc415e790 2024-04-15T15:16:34.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.810+0000 7f1dc0ff9640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1da406c8f0 0x7f1da406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.811+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1dbc0aede0 con 0x7f1dc415e790 2024-04-15T15:16:34.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.811+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1da406c8f0 0x7f1da406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.811+0000 7f1dc2ffd640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1da406c8f0 0x7f1da406edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1db4006fd0 tx=0x7f1db4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.815+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1dbc07ce90 con 0x7f1dc415e790 2024-04-15T15:16:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.999+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1dc40d1ce0 con 0x7f1dc415e790 2024-04-15T15:16:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:34.999+0000 7f1dc0ff9640 1 -- 172.21.15.46:0/3355204918 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f1dbc07c790 con 0x7f1dc415e790 2024-04-15T15:16:35.001 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1da406c8f0 msgr2=0x7f1da406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1da406c8f0 0x7f1da406edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1db4006fd0 tx=0x7f1db4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 msgr2=0x7f1dc40b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 0x7f1dc40b4b20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc009d90 tx=0x7f1dbc004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 shutdown_connections 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1da406c8f0 0x7f1da406edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dc415e790 0x7f1dc40b4b20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1dc40d1610 0x7f1dc40b12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 --2- 172.21.15.46:0/3355204918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1dc40cfd20 0x7f1dc409e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 >> 172.21.15.46:0/3355204918 conn(0x7f1dc408bf60 msgr2=0x7f1dc40d0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 shutdown_connections 2024-04-15T15:16:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.001+0000 7f1dc984a640 1 -- 172.21.15.46:0/3355204918 wait complete. 2024-04-15T15:16:35.014 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.118+0000 7f0ba4767640 1 Processor -- start 2024-04-15T15:16:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.118+0000 7f0ba4767640 1 -- start start 2024-04-15T15:16:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba4767640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c15e790 0x7f0b9c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba4767640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b9c0d2480 con 0x7f0b9c0d1610 2024-04-15T15:16:35.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba4767640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b9c0d2760 con 0x7f0b9c0cfd20 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba4767640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9c0d2db0 con 0x7f0b9c15e790 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba2d04640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0b9c0d1610 0x7f0b9c0b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33016/0 (socket says 172.21.15.46:33016) 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba2d04640 1 -- 172.21.15.46:0/2335200634 learned_addr learned my addr 172.21.15.46:0/2335200634 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.119+0000 7f0ba2503640 1 --2- 172.21.15.46:0/2335200634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c15e790 0x7f0b9c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba1501640 1 -- 172.21.15.46:0/2335200634 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4041611340 0 0) 0x7f0b9c0d2480 con 0x7f0b9c0d1610 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba1501640 1 -- 172.21.15.46:0/2335200634 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b84003520 con 0x7f0b9c0d1610 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba2503640 1 -- 172.21.15.46:0/2335200634 >> v1:172.21.15.78:6789/0 conn(0x7f0b9c0cfd20 legacy=0x7f0b9c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba1501640 1 -- 172.21.15.46:0/2335200634 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952872753 0 0) 0x7f0b9c0d2760 con 0x7f0b9c0cfd20 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba2503640 1 -- 172.21.15.46:0/2335200634 >> v1:172.21.15.46:6789/0 conn(0x7f0b9c0d1610 legacy=0x7f0b9c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba2503640 1 -- 172.21.15.46:0/2335200634 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9c0d2a40 con 0x7f0b9c15e790 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.120+0000 7f0ba2503640 1 --2- 172.21.15.46:0/2335200634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c15e790 0x7f0b9c0d3240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009dc0 tx=0x7f0b8c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f549e0c3b5631266 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba1501640 1 -- 172.21.15.46:0/2335200634 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c039070 con 0x7f0b9c15e790 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba1501640 1 -- 172.21.15.46:0/2335200634 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c02ccd0 con 0x7f0b9c15e790 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba1501640 1 -- 172.21.15.46:0/2335200634 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c034640 con 0x7f0b9c15e790 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 -- 172.21.15.46:0/2335200634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c15e790 msgr2=0x7f0b9c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 --2- 172.21.15.46:0/2335200634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c15e790 0x7f0b9c0d3240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c009dc0 tx=0x7f0b8c02bf20 comp rx=0 tx=0).stop 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 -- 172.21.15.46:0/2335200634 shutdown_connections 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 --2- 172.21.15.46:0/2335200634 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c15e790 0x7f0b9c0d3240 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 -- 172.21.15.46:0/2335200634 >> 172.21.15.46:0/2335200634 conn(0x7f0b9c08bf60 msgr2=0x7f0b9c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 -- 172.21.15.46:0/2335200634 shutdown_connections 2024-04-15T15:16:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.121+0000 7f0ba4767640 1 -- 172.21.15.46:0/2335200634 wait complete. 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.122+0000 7f0ba4767640 1 Processor -- start 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.122+0000 7f0ba4767640 1 -- start start 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba4767640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b9c0cfd20 0x7f0b9c1f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba4767640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c0d1610 0x7f0b9c0cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba4767640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 0x7f0b9c0ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba4767640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c15ecf0 con 0x7f0b9c15e790 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba4767640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9c15eb70 con 0x7f0b9c0cfd20 2024-04-15T15:16:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba4767640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9c0d0160 con 0x7f0b9c0d1610 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba2d04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 0x7f0b9c0ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba2d04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 0x7f0b9c0ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44004/0 (socket says 172.21.15.46:44004) 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba2d04640 1 -- 172.21.15.46:0/1906026269 learned_addr learned my addr 172.21.15.46:0/1906026269 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba2503640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b9c0cfd20 0x7f0b9c1f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.123+0000 7f0ba1d02640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c0d1610 0x7f0b9c0cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2d04640 1 -- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c0d1610 msgr2=0x7f0b9c0cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2d04640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c0d1610 0x7f0b9c0cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2d04640 1 -- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b9c0cfd20 msgr2=0x7f0b9c1f78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2d04640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b9c0cfd20 0x7f0b9c1f78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2d04640 1 -- 172.21.15.46:0/1906026269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9c09e400 con 0x7f0b9c15e790 2024-04-15T15:16:35.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2503640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b9c0cfd20 0x7f0b9c1f78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba1d02640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c0d1610 0x7f0b9c0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba2d04640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 0x7f0b9c0ca340 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400c430 tx=0x7f0b9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94017070 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b94005ce0 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b94007310 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b9c09e6e0 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.124+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b9c09eca0 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.125+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b9c0735d0 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.125+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b9401a070 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.126+0000 7f0b9affd640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b8806c8f0 0x7f0b8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.126+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0b9408b410 con 0x7f0b9c15e790 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.126+0000 7f0ba2503640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b8806c8f0 0x7f0b8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.127+0000 7f0ba2503640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b8806c8f0 0x7f0b8806edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c036000 tx=0x7f0b8c030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.130+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b9c0735d0 con 0x7f0b9c15e790 2024-04-15T15:16:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.327+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0b9c0689e0 con 0x7f0b9c15e790 2024-04-15T15:16:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.328+0000 7f0b9affd640 1 -- 172.21.15.46:0/1906026269 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f0b94059450 con 0x7f0b9c15e790 2024-04-15T15:16:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b8806c8f0 msgr2=0x7f0b8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b8806c8f0 0x7f0b8806edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c036000 tx=0x7f0b8c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 msgr2=0x7f0b9c0ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 0x7f0b9c0ca340 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0b9400c430 tx=0x7f0b9400c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 shutdown_connections 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b8806c8f0 0x7f0b8806edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b9c15e790 0x7f0b9c0ca340 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b9c0d1610 0x7f0b9c0cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 --2- 172.21.15.46:0/1906026269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b9c0cfd20 0x7f0b9c1f78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 >> 172.21.15.46:0/1906026269 conn(0x7f0b9c08bf60 msgr2=0x7f0b9c0d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 shutdown_connections 2024-04-15T15:16:35.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.330+0000 7f0ba4767640 1 -- 172.21.15.46:0/1906026269 wait complete. 2024-04-15T15:16:35.343 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:16:35.343 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:35.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.465+0000 7f5d6d288640 1 Processor -- start 2024-04-15T15:16:35.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.465+0000 7f5d6d288640 1 -- start start 2024-04-15T15:16:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d6d288640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5d680d1a10 0x7f5d680b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d6d288640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d680d08a0 0x7f5d680b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d6d288640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d6815ba70 con 0x7f5d6815b690 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d6d288640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f5d680d0d50 con 0x7f5d680d1a10 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d6d288640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d680ce5b0 con 0x7f5d680d08a0 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d667fc640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5d680d1a10 0x7f5d680b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d677fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d680d08a0 0x7f5d680b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d66ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5d6815b690 0x7f5d680d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:41496/0 (socket says 172.21.15.46:41496) 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.466+0000 7f5d66ffd640 1 -- 172.21.15.46:0/4104898414 learned_addr learned my addr 172.21.15.46:0/4104898414 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d65ffb640 1 -- 172.21.15.46:0/4104898414 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839785015 0 0) 0x7f5d60001110 con 0x7f5d6815b690 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d65ffb640 1 -- 172.21.15.46:0/4104898414 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d54003520 con 0x7f5d6815b690 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d677fe640 1 -- 172.21.15.46:0/4104898414 >> v2:172.21.15.78:3300/0 conn(0x7f5d680d1a10 msgr2=0x7f5d680b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d677fe640 1 --2- 172.21.15.46:0/4104898414 >> v2:172.21.15.78:3300/0 conn(0x7f5d680d1a10 0x7f5d680b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d677fe640 1 -- 172.21.15.46:0/4104898414 >> v1:172.21.15.78:6789/0 conn(0x7f5d6815b690 legacy=0x7f5d680d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d677fe640 1 -- 172.21.15.46:0/4104898414 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d680ce890 con 0x7f5d680d08a0 2024-04-15T15:16:35.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.467+0000 7f5d667fc640 1 --2- 172.21.15.46:0/4104898414 >> v2:172.21.15.78:3300/0 conn(0x7f5d680d1a10 0x7f5d680b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d677fe640 1 --2- 172.21.15.46:0/4104898414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d680d08a0 0x7f5d680b1660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009dc0 tx=0x7f5d5802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c73ebb68320bc0ed server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d65ffb640 1 -- 172.21.15.46:0/4104898414 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58039070 con 0x7f5d680d08a0 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d65ffb640 1 -- 172.21.15.46:0/4104898414 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5802cc40 con 0x7f5d680d08a0 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d65ffb640 1 -- 172.21.15.46:0/4104898414 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58034440 con 0x7f5d680d08a0 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d6d288640 1 -- 172.21.15.46:0/4104898414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d680d08a0 msgr2=0x7f5d680b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d6d288640 1 --2- 172.21.15.46:0/4104898414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d680d08a0 0x7f5d680b1660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5d58009dc0 tx=0x7f5d5802be90 comp rx=0 tx=0).stop 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d6d288640 1 -- 172.21.15.46:0/4104898414 shutdown_connections 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d6d288640 1 --2- 172.21.15.46:0/4104898414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d680d08a0 0x7f5d680b1660 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.468+0000 7f5d6d288640 1 --2- 172.21.15.46:0/4104898414 >> v2:172.21.15.78:3300/0 conn(0x7f5d680d1a10 0x7f5d680b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.469+0000 7f5d6d288640 1 -- 172.21.15.46:0/4104898414 >> 172.21.15.46:0/4104898414 conn(0x7f5d6809a9d0 msgr2=0x7f5d680b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.469+0000 7f5d6d288640 1 -- 172.21.15.46:0/4104898414 shutdown_connections 2024-04-15T15:16:35.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.469+0000 7f5d6d288640 1 -- 172.21.15.46:0/4104898414 wait complete. 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.470+0000 7f5d6d288640 1 Processor -- start 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.470+0000 7f5d6d288640 1 -- start start 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d6d288640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d680d08a0 0x7f5d680af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d6d288640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 0x7f5d680b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d6d288640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d6815b690 0x7f5d681509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d6d288640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d680b1ba0 con 0x7f5d680d1a10 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d6d288640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5d680d1010 con 0x7f5d680d08a0 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d6d288640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d680b1d20 con 0x7f5d6815b690 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 0x7f5d680b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 0x7f5d680b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44012/0 (socket says 172.21.15.46:44012) 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 -- 172.21.15.46:0/3004573672 learned_addr learned my addr 172.21.15.46:0/3004573672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d677fe640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d6815b690 0x7f5d681509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d66ffd640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d680d08a0 0x7f5d680af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 -- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d6815b690 msgr2=0x7f5d681509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d6815b690 0x7f5d681509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 -- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d680d08a0 msgr2=0x7f5d680af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d680d08a0 0x7f5d680af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 -- 172.21.15.46:0/3004573672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d680b6a30 con 0x7f5d680d1a10 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d667fc640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 0x7f5d680b64f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000ebe0 tx=0x7f5d5000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50017070 con 0x7f5d680d1a10 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.471+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5000cbd0 con 0x7f5d680d1a10 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.472+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d50013470 con 0x7f5d680d1a10 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.472+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d680b6cb0 con 0x7f5d680d1a10 2024-04-15T15:16:35.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.472+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d681501d0 con 0x7f5d680d1a10 2024-04-15T15:16:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.472+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d680690f0 con 0x7f5d680d1a10 2024-04-15T15:16:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.473+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d5001a070 con 0x7f5d680d1a10 2024-04-15T15:16:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.473+0000 7f5d5f7fe640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.473+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5d5008c070 con 0x7f5d680d1a10 2024-04-15T15:16:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.474+0000 7f5d66ffd640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.474+0000 7f5d66ffd640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005880 tx=0x7f5d60009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.478+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d680b6a30 con 0x7f5d680d1a10 2024-04-15T15:16:35.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.665+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d68156520 con 0x7f5d680d1a10 2024-04-15T15:16:35.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.666+0000 7f5d5f7fe640 1 -- 172.21.15.46:0/3004573672 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f5d50059ae0 con 0x7f5d680d1a10 2024-04-15T15:16:35.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:35.668 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 msgr2=0x7f5d4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005880 tx=0x7f5d60009040 comp rx=0 tx=0).stop 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 msgr2=0x7f5d680b64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 0x7f5d680b64f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5d5000ebe0 tx=0x7f5d5000c520 comp rx=0 tx=0).stop 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 shutdown_connections 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5d4806c8f0 0x7f5d4806edb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d6815b690 0x7f5d681509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d680d1a10 0x7f5d680b64f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 --2- 172.21.15.46:0/3004573672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d680d08a0 0x7f5d680af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 >> 172.21.15.46:0/3004573672 conn(0x7f5d6809a9d0 msgr2=0x7f5d680cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 shutdown_connections 2024-04-15T15:16:35.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.668+0000 7f5d6d288640 1 -- 172.21.15.46:0/3004573672 wait complete. 2024-04-15T15:16:35.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:35.681 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.788+0000 7f749e6fb640 1 Processor -- start 2024-04-15T15:16:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.788+0000 7f749e6fb640 1 -- start start 2024-04-15T15:16:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.788+0000 7f749e6fb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 0x7f74980cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.788+0000 7f749e6fb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74980b47a0 con 0x7f74980d2360 2024-04-15T15:16:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.788+0000 7f749e6fb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74980b4a80 con 0x7f749815e700 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.788+0000 7f749e6fb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f74980b53b0 con 0x7f74980d3270 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f74977fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f74980d2360 0x7f74980d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33022/0 (socket says 172.21.15.46:33022) 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f74977fe640 1 -- 172.21.15.46:0/1396385367 learned_addr learned my addr 172.21.15.46:0/1396385367 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f749cc98640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 0x7f74980cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3689234409 0 0) 0x7f74980b47a0 con 0x7f74980d2360 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7480003520 con 0x7f74980d2360 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 665523177 0 0) 0x7f7480003520 con 0x7f74980d2360 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 msgr2=0x7f74980cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 0x7f74980cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 >> v1:172.21.15.78:6789/0 conn(0x7f749815e700 legacy=0x7f74980b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74980b4d60 con 0x7f74980d2360 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f748c001670 con 0x7f74980d2360 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 >> v1:172.21.15.46:6789/0 conn(0x7f74980d2360 legacy=0x7f74980d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74800039e0 0x7f7480003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 0x7f7480004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f74800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74980b53b0 con 0x7f7480004d40 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f748000b9f0 con 0x7f74800039e0 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f748000bbf0 con 0x7f74980d3270 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f749cc98640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f74800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f74977fe640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 0x7f7480004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.789+0000 7f7497fff640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74800039e0 0x7f7480003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.790+0000 7f749cc98640 1 -- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 msgr2=0x7f7480004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.790+0000 7f749cc98640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980d3270 0x7f7480004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.790+0000 7f749cc98640 1 -- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74800039e0 msgr2=0x7f7480003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.790+0000 7f749cc98640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74800039e0 0x7f7480003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.790+0000 7f749cc98640 1 -- 172.21.15.46:0/1396385367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7480003520 con 0x7f7480004d40 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.790+0000 7f749cc98640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f74800051c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7490009fc0 tx=0x7f7480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41f420c8fd949a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490039070 con 0x7f7480004d40 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f749002cc20 con 0x7f7480004d40 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f7496ffd640 1 -- 172.21.15.46:0/1396385367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7490009c10 con 0x7f7480004d40 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749e6fb640 1 -- 172.21.15.46:0/1396385367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 msgr2=0x7f74800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749e6fb640 1 --2- 172.21.15.46:0/1396385367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f74800051c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7490009fc0 tx=0x7f7480005830 comp rx=0 tx=0).stop 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749cc98640 1 -- 172.21.15.46:0/1396385367 reap_dead start 2024-04-15T15:16:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749e6fb640 1 -- 172.21.15.46:0/1396385367 shutdown_connections 2024-04-15T15:16:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749e6fb640 1 -- 172.21.15.46:0/1396385367 >> 172.21.15.46:0/1396385367 conn(0x7f749808bf60 msgr2=0x7f74980d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749e6fb640 1 -- 172.21.15.46:0/1396385367 shutdown_connections 2024-04-15T15:16:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.791+0000 7f749e6fb640 1 -- 172.21.15.46:0/1396385367 wait complete. 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.792+0000 7f749e6fb640 1 Processor -- start 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 -- start start 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f7498151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f749815ba30 0x7f74980b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980ba0e0 0x7f74980b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74980d1d40 con 0x7f7480004d40 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f74980b4450 con 0x7f749815ba30 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f749e6fb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f74980d1ec0 con 0x7f74980ba0e0 2024-04-15T15:16:35.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f7497fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f7498151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f7497fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f7498151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44024/0 (socket says 172.21.15.46:44024) 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.793+0000 7f7497fff640 1 -- 172.21.15.46:0/588053132 learned_addr learned my addr 172.21.15.46:0/588053132 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f749cc98640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980ba0e0 0x7f74980b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f74977fe640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f749815ba30 0x7f74980b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7497fff640 1 -- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980ba0e0 msgr2=0x7f74980b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7497fff640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980ba0e0 0x7f74980b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7497fff640 1 -- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f749815ba30 msgr2=0x7f74980b9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7497fff640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f749815ba30 0x7f74980b9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7497fff640 1 -- 172.21.15.46:0/588053132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74980b8de0 con 0x7f7480004d40 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f749cc98640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980ba0e0 0x7f74980b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7497fff640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f7498151af0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f748800f830 tx=0x7f748800fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488018020 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f748800d8e0 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74980b90c0 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748800dcb0 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.794+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74980b9680 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.795+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74980b9880 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.796+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f748801b020 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.796+0000 7f7494ff9640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f748406c8f0 0x7f748406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.796+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f748808c450 con 0x7f7480004d40 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.796+0000 7f74977fe640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f748406c8f0 0x7f748406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.797+0000 7f74977fe640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f748406c8f0 0x7f748406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f749815f3a0 tx=0x7f748c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.801+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74980b9880 con 0x7f7480004d40 2024-04-15T15:16:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.984+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74980689e0 con 0x7f7480004d40 2024-04-15T15:16:35.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.984+0000 7f7494ff9640 1 -- 172.21.15.46:0/588053132 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7488022020 con 0x7f7480004d40 2024-04-15T15:16:35.986 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f748406c8f0 msgr2=0x7f748406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f748406c8f0 0x7f748406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f749815f3a0 tx=0x7f748c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 msgr2=0x7f7498151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f7498151af0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f748800f830 tx=0x7f748800fcf0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 shutdown_connections 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f748406c8f0 0x7f748406edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f74980ba0e0 0x7f74980b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f749815ba30 0x7f74980b9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 --2- 172.21.15.46:0/588053132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7480004d40 0x7f7498151af0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 >> 172.21.15.46:0/588053132 conn(0x7f749808bf60 msgr2=0x7f74980b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 shutdown_connections 2024-04-15T15:16:35.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:35.986+0000 7f749e6fb640 1 -- 172.21.15.46:0/588053132 wait complete. 2024-04-15T15:16:35.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:36.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.104+0000 7fa3ff6c8640 1 Processor -- start 2024-04-15T15:16:36.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.105+0000 7fa3ff6c8640 1 -- start start 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.105+0000 7fa3ff6c8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa3f809b880 0x7fa3f8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.105+0000 7fa3ff6c8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f80d05e0 con 0x7fa3f8070940 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.105+0000 7fa3ff6c8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3f8071260 con 0x7fa3f80d0da0 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3ff6c8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa3f809bce0 con 0x7fa3f809b880 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3fd464640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa3f8070940 0x7fa3f8160310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33026/0 (socket says 172.21.15.46:33026) 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3fd464640 1 -- 172.21.15.46:0/797479177 learned_addr learned my addr 172.21.15.46:0/797479177 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3fcc63640 1 --2- 172.21.15.46:0/797479177 >> v2:172.21.15.78:3300/0 conn(0x7fa3f809b880 0x7fa3f8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3effff640 1 -- 172.21.15.46:0/797479177 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574738362 0 0) 0x7fa3f80d05e0 con 0x7fa3f8070940 2024-04-15T15:16:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3effff640 1 -- 172.21.15.46:0/797479177 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3e4003520 con 0x7fa3f8070940 2024-04-15T15:16:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3fcc63640 1 -- 172.21.15.46:0/797479177 >> v1:172.21.15.78:6789/0 conn(0x7fa3f80d0da0 legacy=0x7fa3f80d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3effff640 1 -- 172.21.15.46:0/797479177 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104594479 0 0) 0x7fa3f8071260 con 0x7fa3f80d0da0 2024-04-15T15:16:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3fcc63640 1 -- 172.21.15.46:0/797479177 >> v1:172.21.15.46:6789/0 conn(0x7fa3f8070940 legacy=0x7fa3f8160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.106+0000 7fa3fcc63640 1 -- 172.21.15.46:0/797479177 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f815f720 con 0x7fa3f809b880 2024-04-15T15:16:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.107+0000 7fa3fcc63640 1 --2- 172.21.15.46:0/797479177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f809b880 0x7fa3f8070d20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80097a0 tx=0x7fa3e802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=438e581ade3c530d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.107+0000 7fa3effff640 1 -- 172.21.15.46:0/797479177 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8034070 con 0x7fa3f809b880 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.107+0000 7fa3effff640 1 -- 172.21.15.46:0/797479177 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3e8030610 con 0x7fa3f809b880 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.107+0000 7fa3effff640 1 -- 172.21.15.46:0/797479177 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e80309c0 con 0x7fa3f809b880 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/797479177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f809b880 msgr2=0x7fa3f8070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/797479177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f809b880 0x7fa3f8070d20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa3e80097a0 tx=0x7fa3e802b960 comp rx=0 tx=0).stop 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/797479177 shutdown_connections 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/797479177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f809b880 0x7fa3f8070d20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/797479177 >> 172.21.15.46:0/797479177 conn(0x7fa3f80b3f40 msgr2=0x7fa3f80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/797479177 shutdown_connections 2024-04-15T15:16:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.108+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/797479177 wait complete. 2024-04-15T15:16:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.109+0000 7fa3ff6c8640 1 Processor -- start 2024-04-15T15:16:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.109+0000 7fa3ff6c8640 1 -- start start 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f8070940 0x7fa3f8154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3f809b880 0x7fa3f80b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 0x7fa3f8157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f80d0880 con 0x7fa3f80d0da0 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f815fce0 con 0x7fa3f8070940 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa3f80d18e0 con 0x7fa3f809b880 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 0x7fa3f8157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 0x7fa3f8157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44034/0 (socket says 172.21.15.46:44034) 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 -- 172.21.15.46:0/2498770558 learned_addr learned my addr 172.21.15.46:0/2498770558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fcc63640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3f809b880 0x7fa3f80b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fd464640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f8070940 0x7fa3f8154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 -- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3f809b880 msgr2=0x7fa3f80b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3f809b880 0x7fa3f80b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 -- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f8070940 msgr2=0x7fa3f8154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f8070940 0x7fa3f8154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 -- 172.21.15.46:0/2498770558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f80b5ba0 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fd464640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f8070940 0x7fa3f8154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3fdc65640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 0x7fa3f8157e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000c790 tx=0x7fa3f000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0003da0 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f0005070 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f80b5e80 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.110+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0005460 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.111+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f815c350 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.111+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3f815be60 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.112+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa3f0005600 con 0x7fa3f80d0da0 2024-04-15T15:16:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.112+0000 7fa3edffb640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3dc06c8f0 0x7fa3dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.112+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa3f008be20 con 0x7fa3f80d0da0 2024-04-15T15:16:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.112+0000 7fa3fd464640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3dc06c8f0 0x7fa3dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.113+0000 7fa3fd464640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3dc06c8f0 0x7fa3dc06edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4007fd0 tx=0x7fa3f4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.116+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3f0059e50 con 0x7fa3f80d0da0 2024-04-15T15:16:36.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.303+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3f80689e0 con 0x7fa3f80d0da0 2024-04-15T15:16:36.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.304+0000 7fa3edffb640 1 -- 172.21.15.46:0/2498770558 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa3f0059750 con 0x7fa3f80d0da0 2024-04-15T15:16:36.305 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3dc06c8f0 msgr2=0x7fa3dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3dc06c8f0 0x7fa3dc06edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4007fd0 tx=0x7fa3f4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 msgr2=0x7fa3f8157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 0x7fa3f8157e30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa3f000c790 tx=0x7fa3f000cc50 comp rx=0 tx=0).stop 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 shutdown_connections 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3dc06c8f0 0x7fa3dc06edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa3f80d0da0 0x7fa3f8157e30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa3f809b880 0x7fa3f80b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 --2- 172.21.15.46:0/2498770558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3f8070940 0x7fa3f8154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 >> 172.21.15.46:0/2498770558 conn(0x7fa3f80b3f40 msgr2=0x7fa3f8070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 shutdown_connections 2024-04-15T15:16:36.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.306+0000 7fa3ff6c8640 1 -- 172.21.15.46:0/2498770558 wait complete. 2024-04-15T15:16:36.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.424+0000 7fdc2dbcd640 1 Processor -- start 2024-04-15T15:16:36.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.424+0000 7fdc2dbcd640 1 -- start start 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc2dbcd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdc280d2360 0x7fdc280d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc2dbcd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc2815eae0 0x7fdc280d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc2dbcd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc280b47a0 con 0x7fdc2815e700 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc2dbcd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdc280b53b0 con 0x7fdc2815eae0 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc2dbcd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fdc280b4a80 con 0x7fdc280d2360 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc27fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc2815eae0 0x7fdc280d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc27fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc2815eae0 0x7fdc280d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44046/0 (socket says 172.21.15.46:44046) 2024-04-15T15:16:36.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc27fff640 1 -- 172.21.15.46:0/1223018486 learned_addr learned my addr 172.21.15.46:0/1223018486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc1effd640 1 --2- 172.21.15.46:0/1223018486 >> v2:172.21.15.78:3300/0 conn(0x7fdc280d2360 0x7fdc280d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.425+0000 7fdc27fff640 1 -- 172.21.15.46:0/1223018486 >> v2:172.21.15.78:3300/0 conn(0x7fdc280d2360 msgr2=0x7fdc280d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc27fff640 1 --2- 172.21.15.46:0/1223018486 >> v2:172.21.15.78:3300/0 conn(0x7fdc280d2360 0x7fdc280d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc27fff640 1 -- 172.21.15.46:0/1223018486 >> v1:172.21.15.78:6789/0 conn(0x7fdc2815e700 legacy=0x7fdc280b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc27fff640 1 -- 172.21.15.46:0/1223018486 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc280b4d60 con 0x7fdc2815eae0 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc1effd640 1 --2- 172.21.15.46:0/1223018486 >> v2:172.21.15.78:3300/0 conn(0x7fdc280d2360 0x7fdc280d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc27fff640 1 --2- 172.21.15.46:0/1223018486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280d30a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdc2808c7b0 tx=0x7fdc2002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93b75027813ec02 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc26ffd640 1 -- 172.21.15.46:0/1223018486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2002ca80 con 0x7fdc2815eae0 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc26ffd640 1 -- 172.21.15.46:0/1223018486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2002cc20 con 0x7fdc2815eae0 2024-04-15T15:16:36.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.426+0000 7fdc26ffd640 1 -- 172.21.15.46:0/1223018486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200343f0 con 0x7fdc2815eae0 2024-04-15T15:16:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1223018486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 msgr2=0x7fdc280d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1223018486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280d30a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fdc2808c7b0 tx=0x7fdc2002bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1223018486 shutdown_connections 2024-04-15T15:16:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1223018486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280d30a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1223018486 >> v2:172.21.15.78:3300/0 conn(0x7fdc280d2360 0x7fdc280d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1223018486 >> 172.21.15.46:0/1223018486 conn(0x7fdc2808bf60 msgr2=0x7fdc280d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1223018486 shutdown_connections 2024-04-15T15:16:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.427+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1223018486 wait complete. 2024-04-15T15:16:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.428+0000 7fdc2dbcd640 1 Processor -- start 2024-04-15T15:16:36.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.428+0000 7fdc2dbcd640 1 -- start start 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc2dbcd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc280d2360 0x7fdc280cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc2dbcd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc2815e700 0x7fdc280c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc2dbcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc2dbcd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc280d1d40 con 0x7fdc2815eae0 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc2dbcd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdc280b4450 con 0x7fdc280d2360 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc2dbcd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdc280d1ec0 con 0x7fdc2815e700 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc27fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44054/0 (socket says 172.21.15.46:44054) 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.429+0000 7fdc27fff640 1 -- 172.21.15.46:0/1258505369 learned_addr learned my addr 172.21.15.46:0/1258505369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc277fe640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc280d2360 0x7fdc280cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc27fff640 1 -- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc2815e700 msgr2=0x7fdc280c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc1effd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc2815e700 0x7fdc280c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc27fff640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc2815e700 0x7fdc280c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc27fff640 1 -- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc280d2360 msgr2=0x7fdc280cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc27fff640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc280d2360 0x7fdc280cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc27fff640 1 -- 172.21.15.46:0/1258505369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc280b9830 con 0x7fdc2815eae0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc27fff640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280cc180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdc2002c4f0 tx=0x7fdc20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20043020 con 0x7fdc2815eae0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2003e450 con 0x7fdc2815eae0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc2806e420 con 0x7fdc2815eae0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc20009510 con 0x7fdc2815eae0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.430+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc2806e140 con 0x7fdc2815eae0 2024-04-15T15:16:36.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.431+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc280689e0 con 0x7fdc2815eae0 2024-04-15T15:16:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.432+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc20046020 con 0x7fdc2815eae0 2024-04-15T15:16:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.432+0000 7fdc24ff9640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc0806c920 0x7fdc0806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.432+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdc200ae5b0 con 0x7fdc2815eae0 2024-04-15T15:16:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.432+0000 7fdc277fe640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc0806c920 0x7fdc0806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.433+0000 7fdc277fe640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc0806c920 0x7fdc0806ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdc18003e00 tx=0x7fdc18009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.436+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc2004d070 con 0x7fdc2815eae0 2024-04-15T15:16:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.621+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdc280ba500 con 0x7fdc2815eae0 2024-04-15T15:16:36.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.622+0000 7fdc24ff9640 1 -- 172.21.15.46:0/1258505369 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fdc2007c5e0 con 0x7fdc2815eae0 2024-04-15T15:16:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc0806c920 msgr2=0x7fdc0806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc0806c920 0x7fdc0806ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdc18003e00 tx=0x7fdc18009040 comp rx=0 tx=0).stop 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 msgr2=0x7fdc280cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280cc180 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdc2002c4f0 tx=0x7fdc20004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 shutdown_connections 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc0806c920 0x7fdc0806ede0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc2815eae0 0x7fdc280cc180 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc2815e700 0x7fdc280c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 --2- 172.21.15.46:0/1258505369 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc280d2360 0x7fdc280cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 >> 172.21.15.46:0/1258505369 conn(0x7fdc2808bf60 msgr2=0x7fdc280cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 shutdown_connections 2024-04-15T15:16:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.624+0000 7fdc2dbcd640 1 -- 172.21.15.46:0/1258505369 wait complete. 2024-04-15T15:16:36.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:36.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.742+0000 7f1769e35640 1 Processor -- start 2024-04-15T15:16:36.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.742+0000 7f1769e35640 1 -- start start 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f1769e35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f176415e790 0x7f17640b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f1769e35640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f176415eb70 con 0x7f17640d1cf0 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f1769e35640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17640d3c80 con 0x7f17640b7be0 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f1769e35640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f17640d3790 con 0x7f176415e790 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f1763fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f17640d1cf0 0x7f17640d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33028/0 (socket says 172.21.15.46:33028) 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f1763fff640 1 -- 172.21.15.46:0/4223931907 learned_addr learned my addr 172.21.15.46:0/4223931907 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.743+0000 7f17637fe640 1 --2- 172.21.15.46:0/4223931907 >> v2:172.21.15.78:3300/0 conn(0x7f176415e790 0x7f17640b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714032506 0 0) 0x7f176415eb70 con 0x7f17640d1cf0 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f174c003520 con 0x7f17640d1cf0 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1320275479 0 0) 0x7f174c003520 con 0x7f17640d1cf0 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 >> v2:172.21.15.78:3300/0 conn(0x7f176415e790 msgr2=0x7f17640b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 --2- 172.21.15.46:0/4223931907 >> v2:172.21.15.78:3300/0 conn(0x7f176415e790 0x7f17640b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 >> v1:172.21.15.78:6789/0 conn(0x7f17640b7be0 legacy=0x7f17640d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17640d3f60 con 0x7f17640d1cf0 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f175c001a10 con 0x7f17640d1cf0 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 >> v1:172.21.15.46:6789/0 conn(0x7f17640d1cf0 legacy=0x7f17640d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0039b0 0x7f174c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c004800 0x7f174c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0052e0 0x7f174c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17640d3790 con 0x7f174c0052e0 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f174c00c1b0 con 0x7f174c004800 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f174c00c3b0 con 0x7f174c0039b0 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1763fff640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0052e0 0x7f174c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f17637fe640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0039b0 0x7f174c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1762ffd640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c004800 0x7f174c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1763fff640 1 -- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0039b0 msgr2=0x7f174c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1763fff640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0039b0 0x7f174c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1763fff640 1 -- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c004800 msgr2=0x7f174c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1763fff640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f174c004800 0x7f174c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.744+0000 7f1763fff640 1 -- 172.21.15.46:0/4223931907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c003520 con 0x7f174c0052e0 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.745+0000 7f1763fff640 1 -- 172.21.15.46:0/4223931907 reap_dead start 2024-04-15T15:16:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.745+0000 7f1763fff640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0052e0 0x7f174c00b9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f176415ed40 tx=0x7f174c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774025814bbdde80 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.745+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c00a480 con 0x7f174c0052e0 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.745+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f175c034650 con 0x7f174c0052e0 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.745+0000 7f17627fc640 1 -- 172.21.15.46:0/4223931907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c034a40 con 0x7f174c0052e0 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 -- 172.21.15.46:0/4223931907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0052e0 msgr2=0x7f174c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0052e0 0x7f174c00b9c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f176415ed40 tx=0x7f174c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 -- 172.21.15.46:0/4223931907 shutdown_connections 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 --2- 172.21.15.46:0/4223931907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c0052e0 0x7f174c00b9c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 -- 172.21.15.46:0/4223931907 >> 172.21.15.46:0/4223931907 conn(0x7f1764067b40 msgr2=0x7f1764067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 -- 172.21.15.46:0/4223931907 shutdown_connections 2024-04-15T15:16:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.746+0000 7f1769e35640 1 -- 172.21.15.46:0/4223931907 wait complete. 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.747+0000 7f1769e35640 1 Processor -- start 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.747+0000 7f1769e35640 1 -- start start 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1769e35640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0052e0 0x7f17640c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1769e35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f176414e3d0 0x7f17640cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1769e35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 0x7f17640c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1769e35640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17640b76a0 con 0x7f17640d0190 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1769e35640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f17640d45b0 con 0x7f176414e3d0 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1769e35640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f17640b7820 con 0x7f174c0052e0 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 0x7f17640c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 0x7f17640c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44068/0 (socket says 172.21.15.46:44068) 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 -- 172.21.15.46:0/2591921675 learned_addr learned my addr 172.21.15.46:0/2591921675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1762ffd640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f176414e3d0 0x7f17640cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f17637fe640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0052e0 0x7f17640c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 -- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0052e0 msgr2=0x7f17640c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0052e0 0x7f17640c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 -- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f176414e3d0 msgr2=0x7f17640cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f176414e3d0 0x7f17640cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1763fff640 1 -- 172.21.15.46:0/2591921675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17640c8d40 con 0x7f17640d0190 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.748+0000 7f1762ffd640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f176414e3d0 0x7f17640cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.749+0000 7f1763fff640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 0x7f17640c8160 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f175c02cc20 tx=0x7f17640d4130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.749+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c039040 con 0x7f17640d0190 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.749+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f175c043070 con 0x7f17640d0190 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.749+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17640c9020 con 0x7f17640d0190 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.749+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c004d80 con 0x7f17640d0190 2024-04-15T15:16:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.749+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17640c95e0 con 0x7f17640d0190 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.750+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17640689e0 con 0x7f17640d0190 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.750+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f175c034650 con 0x7f17640d0190 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.750+0000 7f174bfff640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f175006c8f0 0x7f175006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.751+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f175c0adc00 con 0x7f17640d0190 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.751+0000 7f17637fe640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f175006c8f0 0x7f175006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.751+0000 7f17637fe640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f175006c8f0 0x7f175006edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1754009980 tx=0x7f1754008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.755+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f175c0b4080 con 0x7f17640d0190 2024-04-15T15:16:36.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.939+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f17640cda70 con 0x7f17640d0190 2024-04-15T15:16:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.939+0000 7f174bfff640 1 -- 172.21.15.46:0/2591921675 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f175c07bc30 con 0x7f17640d0190 2024-04-15T15:16:36.941 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f175006c8f0 msgr2=0x7f175006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f175006c8f0 0x7f175006edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1754009980 tx=0x7f1754008040 comp rx=0 tx=0).stop 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 msgr2=0x7f17640c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 0x7f17640c8160 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f175c02cc20 tx=0x7f17640d4130 comp rx=0 tx=0).stop 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 shutdown_connections 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f175006c8f0 0x7f175006edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17640d0190 0x7f17640c8160 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f176414e3d0 0x7f17640cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 --2- 172.21.15.46:0/2591921675 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f174c0052e0 0x7f17640c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 >> 172.21.15.46:0/2591921675 conn(0x7f1764067b40 msgr2=0x7f17640cb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 shutdown_connections 2024-04-15T15:16:36.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:36.941+0000 7f1769e35640 1 -- 172.21.15.46:0/2591921675 wait complete. 2024-04-15T15:16:36.954 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.059+0000 7f1be8575640 1 Processor -- start 2024-04-15T15:16:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.059+0000 7f1be8575640 1 -- start start 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be8575640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1be009b0e0 0x7f1be0070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be8575640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be006ff30 con 0x7f1be00ce8e0 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be8575640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1be00cecc0 con 0x7f1be0071d90 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be8575640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f1be0070820 con 0x7f1be009b0e0 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be5b10640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1be009b0e0 0x7f1be0070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be5b10640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1be009b0e0 0x7f1be0070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44076/0 (socket says 172.21.15.46:44076) 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be5b10640 1 -- 172.21.15.46:0/4199285138 learned_addr learned my addr 172.21.15.46:0/4199285138 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be6311640 1 --1- 172.21.15.46:0/4199285138 >> v1:172.21.15.46:6789/0 conn(0x7f1be00ce8e0 0x7f1be009a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33042/0 (socket says 172.21.15.46:33042) 2024-04-15T15:16:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be5b10640 1 -- 172.21.15.46:0/4199285138 >> v1:172.21.15.78:6789/0 conn(0x7f1be0071d90 legacy=0x7f1be00cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be5b10640 1 -- 172.21.15.46:0/4199285138 >> v1:172.21.15.46:6789/0 conn(0x7f1be00ce8e0 legacy=0x7f1be009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.060+0000 7f1be5b10640 1 -- 172.21.15.46:0/4199285138 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be009b9d0 con 0x7f1be009b0e0 2024-04-15T15:16:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.061+0000 7f1be5b10640 1 --2- 172.21.15.46:0/4199285138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be009b0e0 0x7f1be0070cb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009870 tx=0x7f1bd002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2d1de2915847bf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.061+0000 7f1be530f640 1 -- 172.21.15.46:0/4199285138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0035020 con 0x7f1be009b0e0 2024-04-15T15:16:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.061+0000 7f1be530f640 1 -- 172.21.15.46:0/4199285138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bd0031870 con 0x7f1be009b0e0 2024-04-15T15:16:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.061+0000 7f1be530f640 1 -- 172.21.15.46:0/4199285138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd0031c40 con 0x7f1be009b0e0 2024-04-15T15:16:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 -- 172.21.15.46:0/4199285138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be009b0e0 msgr2=0x7f1be0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 --2- 172.21.15.46:0/4199285138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be009b0e0 0x7f1be0070cb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0009870 tx=0x7f1bd002b910 comp rx=0 tx=0).stop 2024-04-15T15:16:37.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 -- 172.21.15.46:0/4199285138 shutdown_connections 2024-04-15T15:16:37.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 --2- 172.21.15.46:0/4199285138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be009b0e0 0x7f1be0070cb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 -- 172.21.15.46:0/4199285138 >> 172.21.15.46:0/4199285138 conn(0x7f1be00b3f40 msgr2=0x7f1be00b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:37.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 -- 172.21.15.46:0/4199285138 shutdown_connections 2024-04-15T15:16:37.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.062+0000 7f1be8575640 1 -- 172.21.15.46:0/4199285138 wait complete. 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.063+0000 7f1be8575640 1 Processor -- start 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.063+0000 7f1be8575640 1 -- start start 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 0x7f1be00c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1be009b0e0 0x7f1be015cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00ce8e0 0x7f1be009c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1be00704b0 con 0x7f1be0071d90 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1be00701d0 con 0x7f1be00ce8e0 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1be0071970 con 0x7f1be009b0e0 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 0x7f1be00c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 0x7f1be00c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44088/0 (socket says 172.21.15.46:44088) 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 -- 172.21.15.46:0/1427825985 learned_addr learned my addr 172.21.15.46:0/1427825985 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6b12640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00ce8e0 0x7f1be009c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 -- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1be009b0e0 msgr2=0x7f1be015cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be5b10640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1be009b0e0 0x7f1be015cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1be009b0e0 0x7f1be015cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 -- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00ce8e0 msgr2=0x7f1be009c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00ce8e0 0x7f1be009c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 -- 172.21.15.46:0/1427825985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1be00ca470 con 0x7f1be0071d90 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6b12640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00ce8e0 0x7f1be009c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be6311640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 0x7f1be00c9f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c3c0 tx=0x7f1bdc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc0074e0 con 0x7f1be0071d90 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bdc007680 con 0x7f1be0071d90 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1be006cbc0 con 0x7f1be0071d90 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc004a00 con 0x7f1be0071d90 2024-04-15T15:16:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.064+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be006d180 con 0x7f1be0071d90 2024-04-15T15:16:37.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.065+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be00689e0 con 0x7f1be0071d90 2024-04-15T15:16:37.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.066+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1bdc01a020 con 0x7f1be0071d90 2024-04-15T15:16:37.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.066+0000 7f1bd6ffd640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1bc406c8f0 0x7f1bc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.066+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1bdc08b740 con 0x7f1be0071d90 2024-04-15T15:16:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.067+0000 7f1be5b10640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1bc406c8f0 0x7f1bc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.067+0000 7f1be5b10640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1bc406c8f0 0x7f1bc406edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0036000 tx=0x7f1bd003c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.071+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1be00ca470 con 0x7f1be0071d90 2024-04-15T15:16:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.270+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1be01608b0 con 0x7f1be0071d90 2024-04-15T15:16:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.270+0000 7f1bd6ffd640 1 -- 172.21.15.46:0/1427825985 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f1bdc0596d0 con 0x7f1be0071d90 2024-04-15T15:16:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1bc406c8f0 msgr2=0x7f1bc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1bc406c8f0 0x7f1bc406edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0036000 tx=0x7f1bd003c040 comp rx=0 tx=0).stop 2024-04-15T15:16:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 msgr2=0x7f1be00c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 0x7f1be00c9f30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00c3c0 tx=0x7f1bdc00c880 comp rx=0 tx=0).stop 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 shutdown_connections 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1bc406c8f0 0x7f1bc406edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1be00ce8e0 0x7f1be009c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1be009b0e0 0x7f1be015cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 --2- 172.21.15.46:0/1427825985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1be0071d90 0x7f1be00c9f30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.272+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 >> 172.21.15.46:0/1427825985 conn(0x7f1be00b3f40 msgr2=0x7f1be0070cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.273+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 shutdown_connections 2024-04-15T15:16:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.273+0000 7f1be8575640 1 -- 172.21.15.46:0/1427825985 wait complete. 2024-04-15T15:16:37.285 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-04-15T15:16:37.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:37.320 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:37.321 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:16:37.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.321 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:16:37.444 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:16:37.445 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:37.461 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:16:37.461 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:37.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.659+0000 7f5be3c4a640 1 Processor -- start 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.659+0000 7f5be3c4a640 1 -- start start 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be3c4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc15ebb0 0x7f5bdc06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be3c4a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5bdc09b780 0x7f5bdc0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be3c4a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bdc0cccb0 con 0x7f5bdc09a9d0 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be3c4a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f5bdc0cd8c0 con 0x7f5bdc09b780 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be3c4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5bdc0ced60 con 0x7f5bdc15ebb0 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be19e6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5bdc09a9d0 0x7f5bdc09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33054/0 (socket says 172.21.15.46:33054) 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.660+0000 7f5be19e6640 1 -- 172.21.15.46:0/3298841529 learned_addr learned my addr 172.21.15.46:0/3298841529 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be21e7640 1 --2- 172.21.15.46:0/3298841529 >> v2:172.21.15.78:3300/0 conn(0x7f5bdc09b780 0x7f5bdc0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc15ebb0 0x7f5bdc06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 559378564 0 0) 0x7f5bdc0cccb0 con 0x7f5bdc09a9d0 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bc4003520 con 0x7f5bdc09a9d0 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1638847633 0 0) 0x7f5bc4003520 con 0x7f5bdc09a9d0 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc15ebb0 msgr2=0x7f5bdc06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc15ebb0 0x7f5bdc06cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 >> v2:172.21.15.78:3300/0 conn(0x7f5bdc09b780 msgr2=0x7f5bdc0b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 --2- 172.21.15.46:0/3298841529 >> v2:172.21.15.78:3300/0 conn(0x7f5bdc09b780 0x7f5bdc0b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc0ccf90 con 0x7f5bdc09a9d0 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f5bd4001670 con 0x7f5bdc09a9d0 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 >> v1:172.21.15.46:6789/0 conn(0x7f5bdc09a9d0 legacy=0x7f5bdc09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bc40039b0 0x7f5bc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc4004310 0x7f5bc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004e50 0x7f5bc40052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc0cd8c0 con 0x7f5bc4004310 2024-04-15T15:16:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc0ced60 con 0x7f5bc4004e50 2024-04-15T15:16:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc400bd40 con 0x7f5bc40039b0 2024-04-15T15:16:37.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc4004310 0x7f5bc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be19e6640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bc40039b0 0x7f5bc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be21e7640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004e50 0x7f5bc40052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 -- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bc40039b0 msgr2=0x7f5bc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bc40039b0 0x7f5bc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 -- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004e50 msgr2=0x7f5bc40052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004e50 0x7f5bc40052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.661+0000 7f5be11e5640 1 -- 172.21.15.46:0/3298841529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc4003520 con 0x7f5bc4004310 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.662+0000 7f5be11e5640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc4004310 0x7f5bc4004800 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009ac0 tx=0x7f5bc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726e9009a7ac3cde server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.662+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0039070 con 0x7f5bc4004310 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.662+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd002c6b0 con 0x7f5bc4004310 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.662+0000 7f5be09e4640 1 -- 172.21.15.46:0/3298841529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0009690 con 0x7f5bc4004310 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be3c4a640 1 -- 172.21.15.46:0/3298841529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc4004310 msgr2=0x7f5bc4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/3298841529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc4004310 0x7f5bc4004800 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009ac0 tx=0x7f5bc4005830 comp rx=0 tx=0).stop 2024-04-15T15:16:37.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be21e7640 1 -- 172.21.15.46:0/3298841529 reap_dead start 2024-04-15T15:16:37.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be3c4a640 1 -- 172.21.15.46:0/3298841529 shutdown_connections 2024-04-15T15:16:37.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be3c4a640 1 -- 172.21.15.46:0/3298841529 >> 172.21.15.46:0/3298841529 conn(0x7f5bdc0b3fb0 msgr2=0x7f5bdc0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:37.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be3c4a640 1 -- 172.21.15.46:0/3298841529 shutdown_connections 2024-04-15T15:16:37.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.663+0000 7f5be3c4a640 1 -- 172.21.15.46:0/3298841529 wait complete. 2024-04-15T15:16:37.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.664+0000 7f5be3c4a640 1 Processor -- start 2024-04-15T15:16:37.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.664+0000 7f5be3c4a640 1 -- start start 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be3c4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004310 0x7f5bdc15f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be3c4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc0c8a50 0x7f5bdc0c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be3c4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 0x7f5bdc0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be3c4a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc0b5310 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be3c4a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc0cd550 con 0x7f5bc4004310 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be3c4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5bdc0b5490 con 0x7f5bdc0c8a50 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 0x7f5bdc0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 0x7f5bdc0c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44104/0 (socket says 172.21.15.46:44104) 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 -- 172.21.15.46:0/1839897677 learned_addr learned my addr 172.21.15.46:0/1839897677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 -- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc0c8a50 msgr2=0x7f5bdc0c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc0c8a50 0x7f5bdc0c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 -- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004310 msgr2=0x7f5bdc15f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be19e6640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004310 0x7f5bdc15f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004310 0x7f5bdc15f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be21e7640 1 -- 172.21.15.46:0/1839897677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc0ca8b0 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.665+0000 7f5be19e6640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004310 0x7f5bdc15f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.666+0000 7f5be21e7640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 0x7f5bdc0c6bb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc162400 tx=0x7f5bcc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.666+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00fde0 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.666+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bcc01d070 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.666+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc0cab90 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.666+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc015670 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.666+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc15fe40 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.667+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bdc0b8380 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.667+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5bcc015810 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.668+0000 7f5bda7fc640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5bc806c8f0 0x7f5bc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.668+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f5bcc08c9a0 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.668+0000 7f5be19e6640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5bc806c8f0 0x7f5bc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.669+0000 7f5be19e6640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5bc806c8f0 0x7f5bc806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4001eb0 tx=0x7f5bd4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.672+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5bdc0b8380 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.859+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5bdc068b10 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.860+0000 7f5bda7fc640 1 -- 172.21.15.46:0/1839897677 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f5bdc068b10 con 0x7f5bdc0c8ed0 2024-04-15T15:16:37.862 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:37.862 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5bc806c8f0 msgr2=0x7f5bc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5bc806c8f0 0x7f5bc806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4001eb0 tx=0x7f5bd4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 msgr2=0x7f5bdc0c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 0x7f5bdc0c6bb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc162400 tx=0x7f5bcc00f580 comp rx=0 tx=0).stop 2024-04-15T15:16:37.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 shutdown_connections 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5bc806c8f0 0x7f5bc806edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bdc0c8ed0 0x7f5bdc0c6bb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5bdc0c8a50 0x7f5bdc0c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 --2- 172.21.15.46:0/1839897677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bc4004310 0x7f5bdc15f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 >> 172.21.15.46:0/1839897677 conn(0x7f5bdc0b3fb0 msgr2=0x7f5bdc070680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 shutdown_connections 2024-04-15T15:16:37.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.862+0000 7f5be3c4a640 1 -- 172.21.15.46:0/1839897677 wait complete. 2024-04-15T15:16:37.879 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:37.880 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:37.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.984+0000 7fac9b73d640 1 Processor -- start 2024-04-15T15:16:37.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.985+0000 7fac9b73d640 1 -- start start 2024-04-15T15:16:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.985+0000 7fac9b73d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 0x7fac940cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.985+0000 7fac9b73d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac940b47a0 con 0x7fac9415e700 2024-04-15T15:16:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.985+0000 7fac9b73d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac940b4a80 con 0x7fac940d2360 2024-04-15T15:16:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.985+0000 7fac9b73d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fac940b53b0 con 0x7fac940d3270 2024-04-15T15:16:37.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac994d9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac9415e700 0x7fac940b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33066/0 (socket says 172.21.15.46:33066) 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac994d9640 1 -- 172.21.15.46:0/2108389088 learned_addr learned my addr 172.21.15.46:0/2108389088 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac99cda640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 0x7fac940cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 752113506 0 0) 0x7fac940b47a0 con 0x7fac9415e700 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac80003520 con 0x7fac9415e700 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 36085161 0 0) 0x7fac80003520 con 0x7fac9415e700 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 msgr2=0x7fac940cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 0x7fac940cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 >> v1:172.21.15.78:6789/0 conn(0x7fac940d2360 legacy=0x7fac940d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac940b4d60 con 0x7fac9415e700 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fac900017f0 con 0x7fac9415e700 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 >> v1:172.21.15.46:6789/0 conn(0x7fac9415e700 legacy=0x7fac940b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800039e0 0x7fac80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac80004390 0x7fac80004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 0x7fac800050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fac940b53b0 con 0x7fac80004390 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac8000b9f0 con 0x7fac800039e0 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fac8000bbf0 con 0x7fac940d3270 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac98cd8640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac80004390 0x7fac80004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac994d9640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800039e0 0x7fac80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.986+0000 7fac99cda640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 0x7fac800050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac98cd8640 1 -- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 msgr2=0x7fac800050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac98cd8640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac940d3270 0x7fac800050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac98cd8640 1 -- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800039e0 msgr2=0x7fac80003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac98cd8640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac800039e0 0x7fac80003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac98cd8640 1 -- 172.21.15.46:0/2108389088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac80003520 con 0x7fac80004390 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac98cd8640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac80004390 0x7fac80004860 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fac84001910 tx=0x7fac80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e96532e642bbfb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac840014e0 con 0x7fac80004390 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac84037070 con 0x7fac80004390 2024-04-15T15:16:37.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.987+0000 7fac8bfff640 1 -- 172.21.15.46:0/2108389088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac84032840 con 0x7fac80004390 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac9b73d640 1 -- 172.21.15.46:0/2108389088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac80004390 msgr2=0x7fac80004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac9b73d640 1 --2- 172.21.15.46:0/2108389088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac80004390 0x7fac80004860 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fac84001910 tx=0x7fac80005830 comp rx=0 tx=0).stop 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac99cda640 1 -- 172.21.15.46:0/2108389088 reap_dead start 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac9b73d640 1 -- 172.21.15.46:0/2108389088 shutdown_connections 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac9b73d640 1 -- 172.21.15.46:0/2108389088 >> 172.21.15.46:0/2108389088 conn(0x7fac9408bf60 msgr2=0x7fac940d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac9b73d640 1 -- 172.21.15.46:0/2108389088 shutdown_connections 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.988+0000 7fac9b73d640 1 -- 172.21.15.46:0/2108389088 wait complete. 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.989+0000 7fac9b73d640 1 Processor -- start 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.989+0000 7fac9b73d640 1 -- start start 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac9b73d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac80004390 0x7fac94161a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac9b73d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9415eae0 0x7fac9414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac9b73d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 0x7fac940b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac9b73d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fac940d1d40 con 0x7fac94150420 2024-04-15T15:16:37.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac9b73d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac940b4450 con 0x7fac9415eae0 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac9b73d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fac940d1ec0 con 0x7fac80004390 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 0x7fac940b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 0x7fac940b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44126/0 (socket says 172.21.15.46:44126) 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 -- 172.21.15.46:0/1022310813 learned_addr learned my addr 172.21.15.46:0/1022310813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac994d9640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac80004390 0x7fac94161a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac98cd8640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9415eae0 0x7fac9414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 -- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac80004390 msgr2=0x7fac94161a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac80004390 0x7fac94161a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 -- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9415eae0 msgr2=0x7fac9414f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9415eae0 0x7fac9414f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac99cda640 1 -- 172.21.15.46:0/1022310813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9414ff10 con 0x7fac94150420 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.990+0000 7fac98cd8640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9415eae0 0x7fac9414f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.991+0000 7fac99cda640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 0x7fac940b1430 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fac9406c640 tx=0x7fac8c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.991+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c015430 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.991+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8c0155f0 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.991+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0159e0 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.991+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac9415ef60 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.991+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac940cd0c0 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.992+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac940689e0 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.992+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac8c015b80 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.993+0000 7fac89ffb640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac7806c920 0x7fac7806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.993+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fac8c08cdf0 con 0x7fac94150420 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.993+0000 7fac994d9640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac7806c920 0x7fac7806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:37.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.993+0000 7fac994d9640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac7806c920 0x7fac7806ede0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac90000c00 tx=0x7fac90009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:37.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:37.997+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac9414ff10 con 0x7fac94150420 2024-04-15T15:16:38.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.183+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac940ca820 con 0x7fac94150420 2024-04-15T15:16:38.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.183+0000 7fac89ffb640 1 -- 172.21.15.46:0/1022310813 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fac8c019030 con 0x7fac94150420 2024-04-15T15:16:38.185 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:38.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac7806c920 msgr2=0x7fac7806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac7806c920 0x7fac7806ede0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac90000c00 tx=0x7fac90009040 comp rx=0 tx=0).stop 2024-04-15T15:16:38.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 msgr2=0x7fac940b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 0x7fac940b1430 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fac9406c640 tx=0x7fac8c00f580 comp rx=0 tx=0).stop 2024-04-15T15:16:38.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 shutdown_connections 2024-04-15T15:16:38.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac7806c920 0x7fac7806ede0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac94150420 0x7fac940b1430 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9415eae0 0x7fac9414f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 --2- 172.21.15.46:0/1022310813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac80004390 0x7fac94161a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 >> 172.21.15.46:0/1022310813 conn(0x7fac9408bf60 msgr2=0x7fac940d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 shutdown_connections 2024-04-15T15:16:38.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.185+0000 7fac9b73d640 1 -- 172.21.15.46:0/1022310813 wait complete. 2024-04-15T15:16:38.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:38.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.303+0000 7f7a100c5640 1 Processor -- start 2024-04-15T15:16:38.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.304+0000 7f7a100c5640 1 -- start start 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.304+0000 7f7a100c5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a0815c8d0 0x7f7a080cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.304+0000 7f7a100c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7a0809b0e0 0x7f7a080b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.304+0000 7f7a100c5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a080d0350 con 0x7f7a0809b650 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.304+0000 7f7a100c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7a080b5b00 con 0x7f7a0809b0e0 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.304+0000 7f7a100c5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7a080ce100 con 0x7f7a0815c8d0 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7a0809b0e0 0x7f7a080b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7a0809b0e0 0x7f7a080b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44134/0 (socket says 172.21.15.46:44134) 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 -- 172.21.15.46:0/3515397334 learned_addr learned my addr 172.21.15.46:0/3515397334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0de61640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a0815c8d0 0x7f7a080cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 -- 172.21.15.46:0/3515397334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a0815c8d0 msgr2=0x7f7a080cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a0815c8d0 0x7f7a080cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 -- 172.21.15.46:0/3515397334 >> v1:172.21.15.78:6789/0 conn(0x7f7a0809b650 legacy=0x7f7a0809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0e662640 1 -- 172.21.15.46:0/3515397334 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a080b54b0 con 0x7f7a0809b0e0 2024-04-15T15:16:38.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.305+0000 7f7a0de61640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a0815c8d0 0x7f7a080cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:38.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a0e662640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b0e0 0x7f7a080b5f90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009e50 tx=0x7f7a0002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fe662ca9d2f8a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a0ce5f640 1 -- 172.21.15.46:0/3515397334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0002c9e0 con 0x7f7a0809b0e0 2024-04-15T15:16:38.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a0ce5f640 1 -- 172.21.15.46:0/3515397334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0002cb80 con 0x7f7a0809b0e0 2024-04-15T15:16:38.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a0ce5f640 1 -- 172.21.15.46:0/3515397334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a000343f0 con 0x7f7a0809b0e0 2024-04-15T15:16:38.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 -- 172.21.15.46:0/3515397334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b0e0 msgr2=0x7f7a080b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b0e0 0x7f7a080b5f90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009e50 tx=0x7f7a0002be40 comp rx=0 tx=0).stop 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 -- 172.21.15.46:0/3515397334 shutdown_connections 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b0e0 0x7f7a080b5f90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 --2- 172.21.15.46:0/3515397334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a0815c8d0 0x7f7a080cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 -- 172.21.15.46:0/3515397334 >> 172.21.15.46:0/3515397334 conn(0x7f7a080b3fb0 msgr2=0x7f7a080b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 -- 172.21.15.46:0/3515397334 shutdown_connections 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.306+0000 7f7a100c5640 1 -- 172.21.15.46:0/3515397334 wait complete. 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 Processor -- start 2024-04-15T15:16:38.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 -- start start 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a0809b0e0 0x7f7a080c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 0x7f7a0815a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a080c8930 0x7f7a0815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7a080b6650 con 0x7f7a0809b650 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7a080b64d0 con 0x7f7a0809b0e0 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a100c5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7a080b67d0 con 0x7f7a080c8930 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a0d660640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 0x7f7a0815a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a0d660640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 0x7f7a0815a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44150/0 (socket says 172.21.15.46:44150) 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.308+0000 7f7a0d660640 1 -- 172.21.15.46:0/1336070538 learned_addr learned my addr 172.21.15.46:0/1336070538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0e662640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a080c8930 0x7f7a0815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0de61640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a0809b0e0 0x7f7a080c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0d660640 1 -- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a080c8930 msgr2=0x7f7a0815ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0d660640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a080c8930 0x7f7a0815ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0d660640 1 -- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a0809b0e0 msgr2=0x7f7a080c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0d660640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a0809b0e0 0x7f7a080c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0d660640 1 -- 172.21.15.46:0/1336070538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a080c8400 con 0x7f7a0809b650 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0de61640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a0809b0e0 0x7f7a080c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a0d660640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 0x7f7a0815a5c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f79f800c460 tx=0x7f79f800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8016020 con 0x7f7a0809b650 2024-04-15T15:16:38.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f8012870 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a081f3d10 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f8012c60 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.309+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a081f42d0 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.310+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a080690f0 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.311+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79f8019020 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.311+0000 7f79fe7fc640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79ec06c8f0 0x7f79ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.311+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f79f808bc70 con 0x7f7a0809b650 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.311+0000 7f7a0de61640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79ec06c8f0 0x7f79ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.312+0000 7f7a0de61640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79ec06c8f0 0x7f79ec06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400a990 tx=0x7f7a04008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.315+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79f8020030 con 0x7f7a0809b650 2024-04-15T15:16:38.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.501+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a080c8630 con 0x7f7a0809b650 2024-04-15T15:16:38.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.501+0000 7f79fe7fc640 1 -- 172.21.15.46:0/1336070538 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7a080c8400 con 0x7f7a0809b650 2024-04-15T15:16:38.504 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.503+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79ec06c8f0 msgr2=0x7f79ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.503+0000 7f7a100c5640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79ec06c8f0 0x7f79ec06edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400a990 tx=0x7f7a04008040 comp rx=0 tx=0).stop 2024-04-15T15:16:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.503+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 msgr2=0x7f7a0815a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.503+0000 7f7a100c5640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 0x7f7a0815a5c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f79f800c460 tx=0x7f79f800c920 comp rx=0 tx=0).stop 2024-04-15T15:16:38.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 shutdown_connections 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79ec06c8f0 0x7f79ec06edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a080c8930 0x7f7a0815ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a0809b650 0x7f7a0815a5c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 --2- 172.21.15.46:0/1336070538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a0809b0e0 0x7f7a080c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 >> 172.21.15.46:0/1336070538 conn(0x7f7a080b3fb0 msgr2=0x7f7a0806fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 shutdown_connections 2024-04-15T15:16:38.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.504+0000 7f7a100c5640 1 -- 172.21.15.46:0/1336070538 wait complete. 2024-04-15T15:16:38.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.624+0000 7f232a893640 1 Processor -- start 2024-04-15T15:16:38.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.624+0000 7f232a893640 1 -- start start 2024-04-15T15:16:38.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f232a893640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 0x7f23240b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f232a893640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f23240d1cf0 0x7f23240d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f232a893640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f232415eb70 con 0x7f23240b7be0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f232a893640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f23240d3790 con 0x7f23240d1cf0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f232a893640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f23240d45b0 con 0x7f232415e790 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f23237fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f23240b7be0 0x7f23240d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33074/0 (socket says 172.21.15.46:33074) 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f23237fe640 1 -- 172.21.15.46:0/817439998 learned_addr learned my addr 172.21.15.46:0/817439998 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2323fff640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 0x7f23240b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2145921979 0 0) 0x7f232415eb70 con 0x7f23240b7be0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2328e30640 1 --2- 172.21.15.46:0/817439998 >> v2:172.21.15.78:3300/0 conn(0x7f23240d1cf0 0x7f23240d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f230c003520 con 0x7f23240b7be0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2986758020 0 0) 0x7f230c003520 con 0x7f23240b7be0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 msgr2=0x7f23240b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 0x7f23240b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 >> v2:172.21.15.78:3300/0 conn(0x7f23240d1cf0 msgr2=0x7f23240d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 --2- 172.21.15.46:0/817439998 >> v2:172.21.15.78:3300/0 conn(0x7f23240d1cf0 0x7f23240d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23240d3c80 con 0x7f23240b7be0 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f2318001690 con 0x7f23240b7be0 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 >> v1:172.21.15.46:6789/0 conn(0x7f23240b7be0 legacy=0x7f23240d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 0x7f230c0039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.625+0000 7f2322ffd640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f230c003fa0 0x7f230c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f2322ffd640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c004d40 0x7f230c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f23240d3790 con 0x7f230c003fa0 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f23240d45b0 con 0x7f230c004d40 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f230c00bb20 con 0x7f232415e790 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f23237fe640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f230c003fa0 0x7f230c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f2323fff640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 0x7f230c0039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f2328e30640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c004d40 0x7f230c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f23237fe640 1 -- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 msgr2=0x7f230c0039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f23237fe640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f232415e790 0x7f230c0039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f23237fe640 1 -- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c004d40 msgr2=0x7f230c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f23237fe640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c004d40 0x7f230c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.626+0000 7f23237fe640 1 -- 172.21.15.46:0/817439998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230c003520 con 0x7f230c003fa0 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f23237fe640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f230c003fa0 0x7f230c004800 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f231800c160 tx=0x7f230c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a2a6b21e519a38 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318007c80 con 0x7f230c003fa0 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2318030070 con 0x7f230c003fa0 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f2322ffd640 1 -- 172.21.15.46:0/817439998 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318005d10 con 0x7f230c003fa0 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f232a893640 1 -- 172.21.15.46:0/817439998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f230c003fa0 msgr2=0x7f230c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f232a893640 1 --2- 172.21.15.46:0/817439998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f230c003fa0 0x7f230c004800 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f231800c160 tx=0x7f230c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f2328e30640 1 -- 172.21.15.46:0/817439998 reap_dead start 2024-04-15T15:16:38.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f232a893640 1 -- 172.21.15.46:0/817439998 shutdown_connections 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f232a893640 1 -- 172.21.15.46:0/817439998 >> 172.21.15.46:0/817439998 conn(0x7f2324067b40 msgr2=0x7f2324067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f232a893640 1 -- 172.21.15.46:0/817439998 shutdown_connections 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.627+0000 7f232a893640 1 -- 172.21.15.46:0/817439998 wait complete. 2024-04-15T15:16:38.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 Processor -- start 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 -- start start 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c003fa0 0x7f232415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 0x7f23241f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f23240af4c0 0x7f23240af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f23240b76a0 con 0x7f23240aef70 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f23240d4240 con 0x7f230c003fa0 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.629+0000 7f232a893640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f23240b7820 con 0x7f23240af4c0 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 0x7f23241f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 0x7f23241f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44156/0 (socket says 172.21.15.46:44156) 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 -- 172.21.15.46:0/613902672 learned_addr learned my addr 172.21.15.46:0/613902672 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f2328e30640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f23240af4c0 0x7f23240af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f2323fff640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c003fa0 0x7f232415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 -- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f23240af4c0 msgr2=0x7f23240af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f23240af4c0 0x7f23240af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 -- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c003fa0 msgr2=0x7f232415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c003fa0 0x7f232415d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 -- 172.21.15.46:0/613902672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232415df30 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f2323fff640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c003fa0 0x7f232415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f23237fe640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 0x7f23241f52c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f231800c280 tx=0x7f23240d3e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318045070 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2318004bf0 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23240ac810 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318007c80 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.630+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23240acdd0 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.631+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23240689e0 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.632+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2318007070 con 0x7f23240aef70 2024-04-15T15:16:38.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.632+0000 7f2320ff9640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f231006c8f0 0x7f231006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.632+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f23180acd50 con 0x7f23240aef70 2024-04-15T15:16:38.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.633+0000 7f2323fff640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f231006c8f0 0x7f231006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.633+0000 7f2323fff640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f231006c8f0 0x7f231006edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23241f7650 tx=0x7f2314008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.637+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f231807a8a0 con 0x7f23240aef70 2024-04-15T15:16:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.823+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f23240b72a0 con 0x7f23240aef70 2024-04-15T15:16:38.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.824+0000 7f2320ff9640 1 -- 172.21.15.46:0/613902672 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f231807eba0 con 0x7f23240aef70 2024-04-15T15:16:38.826 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:38.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f231006c8f0 msgr2=0x7f231006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f231006c8f0 0x7f231006edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f23241f7650 tx=0x7f2314008040 comp rx=0 tx=0).stop 2024-04-15T15:16:38.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 msgr2=0x7f23241f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 0x7f23241f52c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f231800c280 tx=0x7f23240d3e50 comp rx=0 tx=0).stop 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 shutdown_connections 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f231006c8f0 0x7f231006edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f23240af4c0 0x7f23240af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23240aef70 0x7f23241f52c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 --2- 172.21.15.46:0/613902672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f230c003fa0 0x7f232415d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 >> 172.21.15.46:0/613902672 conn(0x7f2324067b40 msgr2=0x7f2324161500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 shutdown_connections 2024-04-15T15:16:38.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.826+0000 7f232a893640 1 -- 172.21.15.46:0/613902672 wait complete. 2024-04-15T15:16:38.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.944+0000 7f3ecad8d640 1 Processor -- start 2024-04-15T15:16:38.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.944+0000 7f3ecad8d640 1 -- start start 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ecad8d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3ec415e790 0x7f3ec40d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ecad8d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec40d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ecad8d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ec40d0f00 0x7f3ec40d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ecad8d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec40d2db0 con 0x7f3ec40d0f00 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ecad8d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec40d2480 con 0x7f3ec415e790 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ecad8d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec40d2760 con 0x7f3ec40cfd20 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ec932a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ec40d0f00 0x7f3ec40d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ec932a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ec40d0f00 0x7f3ec40d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44160/0 (socket says 172.21.15.46:44160) 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ec932a640 1 -- 172.21.15.46:0/1251304037 learned_addr learned my addr 172.21.15.46:0/1251304037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:38.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ec3fff640 1 --2- 172.21.15.46:0/1251304037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec40d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.945+0000 7f3ec8b29640 1 --2- 172.21.15.46:0/1251304037 >> v2:172.21.15.78:3300/0 conn(0x7f3ec415e790 0x7f3ec40d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.946+0000 7f3ec932a640 1 -- 172.21.15.46:0/1251304037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 msgr2=0x7f3ec40d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.946+0000 7f3ec932a640 1 --2- 172.21.15.46:0/1251304037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec40d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.946+0000 7f3ec932a640 1 -- 172.21.15.46:0/1251304037 >> v2:172.21.15.78:3300/0 conn(0x7f3ec415e790 msgr2=0x7f3ec40d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.946+0000 7f3ec932a640 1 --2- 172.21.15.46:0/1251304037 >> v2:172.21.15.78:3300/0 conn(0x7f3ec415e790 0x7f3ec40d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.946+0000 7f3ec932a640 1 -- 172.21.15.46:0/1251304037 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec40d2a40 con 0x7f3ec40d0f00 2024-04-15T15:16:38.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.946+0000 7f3ec932a640 1 --2- 172.21.15.46:0/1251304037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec40d0f00 0x7f3ec40d13b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40d4600 tx=0x7f3eb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a6b48e9296261 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ec37fe640 1 -- 172.21.15.46:0/1251304037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb402ca70 con 0x7f3ec40d0f00 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ec37fe640 1 -- 172.21.15.46:0/1251304037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb402cc10 con 0x7f3ec40d0f00 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ec37fe640 1 -- 172.21.15.46:0/1251304037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb40343f0 con 0x7f3ec40d0f00 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 -- 172.21.15.46:0/1251304037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec40d0f00 msgr2=0x7f3ec40d13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/1251304037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec40d0f00 0x7f3ec40d13b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40d4600 tx=0x7f3eb402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 -- 172.21.15.46:0/1251304037 shutdown_connections 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/1251304037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec40d0f00 0x7f3ec40d13b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/1251304037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec40d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/1251304037 >> v2:172.21.15.78:3300/0 conn(0x7f3ec415e790 0x7f3ec40d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 -- 172.21.15.46:0/1251304037 >> 172.21.15.46:0/1251304037 conn(0x7f3ec408bf60 msgr2=0x7f3ec40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 -- 172.21.15.46:0/1251304037 shutdown_connections 2024-04-15T15:16:38.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.947+0000 7f3ecad8d640 1 -- 172.21.15.46:0/1251304037 wait complete. 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.948+0000 7f3ecad8d640 1 Processor -- start 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.948+0000 7f3ecad8d640 1 -- start start 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ecad8d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ecad8d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec40d0f00 0x7f3ec4158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ecad8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 0x7f3ec415be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ecad8d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec415ecf0 con 0x7f3ec415e790 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ecad8d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec415eb70 con 0x7f3ec40d0f00 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ecad8d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec40b74f0 con 0x7f3ec40cfd20 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ec932a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 0x7f3ec415be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ec932a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 0x7f3ec415be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44162/0 (socket says 172.21.15.46:44162) 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.949+0000 7f3ec932a640 1 -- 172.21.15.46:0/3141459921 learned_addr learned my addr 172.21.15.46:0/3141459921 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec8b29640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec3fff640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec40d0f00 0x7f3ec4158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec932a640 1 -- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 msgr2=0x7f3ec415a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec932a640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec415a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec932a640 1 -- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec40d0f00 msgr2=0x7f3ec4158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec932a640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec40d0f00 0x7f3ec4158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec932a640 1 -- 172.21.15.46:0/3141459921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec4159660 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec8b29640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec932a640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 0x7f3ec415be80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4036040 tx=0x7f3eb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4043020 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb403e450 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4009510 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec4159940 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.950+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec415a1e0 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.951+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec4159d30 con 0x7f3ec415e790 2024-04-15T15:16:38.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.952+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3eb4046020 con 0x7f3ec415e790 2024-04-15T15:16:38.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.952+0000 7f3ec17fa640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3ea406c8f0 0x7f3ea406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.952+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3eb40ae660 con 0x7f3ec415e790 2024-04-15T15:16:38.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.952+0000 7f3ec8b29640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3ea406c8f0 0x7f3ea406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.953+0000 7f3ec8b29640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3ea406c8f0 0x7f3ea406edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009a20 tx=0x7f3ebc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:38.956+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3eb404d030 con 0x7f3ec415e790 2024-04-15T15:16:39.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.142+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3ec40689e0 con 0x7f3ec415e790 2024-04-15T15:16:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.143+0000 7f3ec17fa640 1 -- 172.21.15.46:0/3141459921 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3eb407c690 con 0x7f3ec415e790 2024-04-15T15:16:39.145 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3ea406c8f0 msgr2=0x7f3ea406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3ea406c8f0 0x7f3ea406edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009a20 tx=0x7f3ebc008040 comp rx=0 tx=0).stop 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 msgr2=0x7f3ec415be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 0x7f3ec415be80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4036040 tx=0x7f3eb4004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 shutdown_connections 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3ea406c8f0 0x7f3ea406edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ec415e790 0x7f3ec415be80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ec40d0f00 0x7f3ec4158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 --2- 172.21.15.46:0/3141459921 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ec40cfd20 0x7f3ec415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 >> 172.21.15.46:0/3141459921 conn(0x7f3ec408bf60 msgr2=0x7f3ec40d3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 shutdown_connections 2024-04-15T15:16:39.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.145+0000 7f3ecad8d640 1 -- 172.21.15.46:0/3141459921 wait complete. 2024-04-15T15:16:39.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.266+0000 7f2c0dce4640 1 Processor -- start 2024-04-15T15:16:39.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.266+0000 7f2c0dce4640 1 -- start start 2024-04-15T15:16:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c0dce4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 0x7f2c080b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c0dce4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c080d3270 0x7f2c080cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c0dce4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c080b47a0 con 0x7f2c080d2360 2024-04-15T15:16:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c0dce4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2c080b53b0 con 0x7f2c080d3270 2024-04-15T15:16:39.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c0dce4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2c080b4a80 con 0x7f2c0815e700 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c077fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 0x7f2c080b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c07fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2c080d3270 0x7f2c080cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2bfffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2c080d2360 0x7f2c080d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:41582/0 (socket says 172.21.15.46:41582) 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c077fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 0x7f2c080b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:51772/0 (socket says 172.21.15.46:51772) 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2bfffff640 1 -- 172.21.15.46:0/4270730375 learned_addr learned my addr 172.21.15.46:0/4270730375 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c077fe640 1 -- 172.21.15.46:0/4270730375 >> v2:172.21.15.78:3300/0 conn(0x7f2c080d3270 msgr2=0x7f2c080cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c077fe640 1 --2- 172.21.15.46:0/4270730375 >> v2:172.21.15.78:3300/0 conn(0x7f2c080d3270 0x7f2c080cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c077fe640 1 -- 172.21.15.46:0/4270730375 >> v1:172.21.15.78:6789/0 conn(0x7f2c080d2360 legacy=0x7f2c080d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c06ffd640 1 -- 172.21.15.46:0/4270730375 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234423448 0 0) 0x7f2c080b47a0 con 0x7f2c080d2360 2024-04-15T15:16:39.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.267+0000 7f2c077fe640 1 -- 172.21.15.46:0/4270730375 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c080b4d60 con 0x7f2c0815e700 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.268+0000 7f2c077fe640 1 --2- 172.21.15.46:0/4270730375 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 0x7f2c080b5840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009790 tx=0x7f2bf802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=def687a89ab2270c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.268+0000 7f2c06ffd640 1 -- 172.21.15.46:0/4270730375 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8035020 con 0x7f2c0815e700 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.268+0000 7f2c06ffd640 1 -- 172.21.15.46:0/4270730375 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf8031890 con 0x7f2c0815e700 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.268+0000 7f2c06ffd640 1 -- 172.21.15.46:0/4270730375 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8031c60 con 0x7f2c0815e700 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 -- 172.21.15.46:0/4270730375 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 msgr2=0x7f2c080b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/4270730375 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 0x7f2c080b5840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8009790 tx=0x7f2bf802b930 comp rx=0 tx=0).stop 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 -- 172.21.15.46:0/4270730375 shutdown_connections 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/4270730375 >> v2:172.21.15.78:3300/0 conn(0x7f2c080d3270 0x7f2c080cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/4270730375 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c0815e700 0x7f2c080b5840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 -- 172.21.15.46:0/4270730375 >> 172.21.15.46:0/4270730375 conn(0x7f2c0808bf60 msgr2=0x7f2c080d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 -- 172.21.15.46:0/4270730375 shutdown_connections 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.269+0000 7f2c0dce4640 1 -- 172.21.15.46:0/4270730375 wait complete. 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.270+0000 7f2c0dce4640 1 Processor -- start 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.270+0000 7f2c0dce4640 1 -- start start 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c0dce4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c080d2360 0x7f2c08159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c0dce4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c080d3270 0x7f2c080cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c0dce4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 0x7f2c080b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c0dce4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2c080b5d80 con 0x7f2c0815e700 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c0dce4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2c080b4450 con 0x7f2c080d2360 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c0dce4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2c080d1d40 con 0x7f2c080d3270 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 0x7f2c080b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c07fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 0x7f2c080b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44170/0 (socket says 172.21.15.46:44170) 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c07fff640 1 -- 172.21.15.46:0/3695288557 learned_addr learned my addr 172.21.15.46:0/3695288557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2bfffff640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c080d3270 0x7f2c080cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.271+0000 7f2c077fe640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c080d2360 0x7f2c08159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c07fff640 1 -- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c080d3270 msgr2=0x7f2c080cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c07fff640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c080d3270 0x7f2c080cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c07fff640 1 -- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c080d2360 msgr2=0x7f2c08159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c07fff640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c080d2360 0x7f2c08159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c07fff640 1 -- 172.21.15.46:0/3695288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c080b3d50 con 0x7f2c0815e700 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c077fe640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c080d2360 0x7f2c08159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c07fff640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 0x7f2c080b3290 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000efc0 tx=0x7f2c0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c00018020 con 0x7f2c0815e700 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c00014480 con 0x7f2c0815e700 2024-04-15T15:16:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c080cc520 con 0x7f2c0815e700 2024-04-15T15:16:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c00014870 con 0x7f2c0815e700 2024-04-15T15:16:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.272+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c080ccae0 con 0x7f2c0815e700 2024-04-15T15:16:39.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.273+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c080ccce0 con 0x7f2c0815e700 2024-04-15T15:16:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.274+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c0001b020 con 0x7f2c0815e700 2024-04-15T15:16:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.274+0000 7f2c04ff9640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2be806c8f0 0x7f2be806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.274+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2c0008cdc0 con 0x7f2c0815e700 2024-04-15T15:16:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.275+0000 7f2c077fe640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2be806c8f0 0x7f2be806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.275+0000 7f2c077fe640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2be806c8f0 0x7f2be806edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8036000 tx=0x7f2bf8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.279+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c080ccce0 con 0x7f2c0815e700 2024-04-15T15:16:39.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.477+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2c080689e0 con 0x7f2c0815e700 2024-04-15T15:16:39.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.477+0000 7f2c04ff9640 1 -- 172.21.15.46:0/3695288557 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f2c00022020 con 0x7f2c0815e700 2024-04-15T15:16:39.479 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2be806c8f0 msgr2=0x7f2be806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2be806c8f0 0x7f2be806edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8036000 tx=0x7f2bf8008030 comp rx=0 tx=0).stop 2024-04-15T15:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 msgr2=0x7f2c080b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 0x7f2c080b3290 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2c0000efc0 tx=0x7f2c0000c490 comp rx=0 tx=0).stop 2024-04-15T15:16:39.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 shutdown_connections 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2be806c8f0 0x7f2be806edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c0815e700 0x7f2c080b3290 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2c080d3270 0x7f2c080cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 --2- 172.21.15.46:0/3695288557 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c080d2360 0x7f2c08159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 >> 172.21.15.46:0/3695288557 conn(0x7f2c0808bf60 msgr2=0x7f2c0815ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 shutdown_connections 2024-04-15T15:16:39.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.479+0000 7f2c0dce4640 1 -- 172.21.15.46:0/3695288557 wait complete. 2024-04-15T15:16:39.493 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:16:39.493 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:39.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:39.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.655+0000 7fe383f32640 1 Processor -- start 2024-04-15T15:16:39.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.655+0000 7fe383f32640 1 -- start start 2024-04-15T15:16:39.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe383f32640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe37c0d0690 0x7fe37c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe383f32640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe37c0b3160 0x7fe37c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe383f32640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37c0d0fb0 con 0x7fe37c15ef70 2024-04-15T15:16:39.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe383f32640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe37c15cb50 con 0x7fe37c0d0690 2024-04-15T15:16:39.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe383f32640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe37c15bfd0 con 0x7fe37c0b3160 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe37c0d0690 0x7fe37c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe37c0d0690 0x7fe37c0d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41298/0 (socket says 172.21.15.46:41298) 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 -- 172.21.15.46:0/1054119632 learned_addr learned my addr 172.21.15.46:0/1054119632 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe3824cf640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe37c15ef70 0x7fe37c15f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40360/0 (socket says 172.21.15.46:40360) 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe3814cd640 1 --2- 172.21.15.46:0/1054119632 >> v2:172.21.15.78:3300/0 conn(0x7fe37c0b3160 0x7fe37c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 -- 172.21.15.46:0/1054119632 >> v2:172.21.15.78:3300/0 conn(0x7fe37c0b3160 msgr2=0x7fe37c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 --2- 172.21.15.46:0/1054119632 >> v2:172.21.15.78:3300/0 conn(0x7fe37c0b3160 0x7fe37c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 -- 172.21.15.46:0/1054119632 >> v1:172.21.15.46:6789/0 conn(0x7fe37c15ef70 legacy=0x7fe37c15f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.656+0000 7fe381cce640 1 -- 172.21.15.46:0/1054119632 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c15c2b0 con 0x7fe37c0d0690 2024-04-15T15:16:39.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.657+0000 7fe381cce640 1 --2- 172.21.15.46:0/1054119632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0d0690 0x7fe37c0d0a70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe37c06fec0 tx=0x7fe37802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d04a49923943b6b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.657+0000 7fe380ccc640 1 -- 172.21.15.46:0/1054119632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37803b070 con 0x7fe37c0d0690 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.657+0000 7fe380ccc640 1 -- 172.21.15.46:0/1054119632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe378036440 con 0x7fe37c0d0690 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.657+0000 7fe380ccc640 1 -- 172.21.15.46:0/1054119632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378036810 con 0x7fe37c0d0690 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 -- 172.21.15.46:0/1054119632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0d0690 msgr2=0x7fe37c0d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 --2- 172.21.15.46:0/1054119632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0d0690 0x7fe37c0d0a70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe37c06fec0 tx=0x7fe37802e170 comp rx=0 tx=0).stop 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 -- 172.21.15.46:0/1054119632 shutdown_connections 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 --2- 172.21.15.46:0/1054119632 >> v2:172.21.15.78:3300/0 conn(0x7fe37c0b3160 0x7fe37c15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 --2- 172.21.15.46:0/1054119632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c0d0690 0x7fe37c0d0a70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 -- 172.21.15.46:0/1054119632 >> 172.21.15.46:0/1054119632 conn(0x7fe37c06be00 msgr2=0x7fe37c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 -- 172.21.15.46:0/1054119632 shutdown_connections 2024-04-15T15:16:39.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.658+0000 7fe383f32640 1 -- 172.21.15.46:0/1054119632 wait complete. 2024-04-15T15:16:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.659+0000 7fe383f32640 1 Processor -- start 2024-04-15T15:16:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.659+0000 7fe383f32640 1 -- start start 2024-04-15T15:16:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe383f32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0b3160 0x7fe37c14c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe383f32640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d0690 0x7fe37c1594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe383f32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 0x7fe37c155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe383f32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c0b4fb0 con 0x7fe37c15ef70 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe383f32640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe37c15c870 con 0x7fe37c0d0690 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe383f32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe37c15bbd0 con 0x7fe37c0b3160 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe3824cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 0x7fe37c155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe3824cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 0x7fe37c155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41300/0 (socket says 172.21.15.46:41300) 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.660+0000 7fe3824cf640 1 -- 172.21.15.46:0/1317302204 learned_addr learned my addr 172.21.15.46:0/1317302204 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe381cce640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0b3160 0x7fe37c14c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3814cd640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d0690 0x7fe37c1594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3824cf640 1 -- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0b3160 msgr2=0x7fe37c14c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3824cf640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0b3160 0x7fe37c14c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3824cf640 1 -- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d0690 msgr2=0x7fe37c1594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3824cf640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d0690 0x7fe37c1594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3824cf640 1 -- 172.21.15.46:0/1317302204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37c159d20 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe381cce640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0b3160 0x7fe37c14c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3814cd640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d0690 0x7fe37c1594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3824cf640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 0x7fe37c155c90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe37400c780 tx=0x7fe37400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe374017070 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe37400a5e0 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37c15a000 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe37400a9d0 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.661+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe37c0cd1b0 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.662+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe37c0cd3b0 con 0x7fe37c15ef70 2024-04-15T15:16:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.663+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe37401a070 con 0x7fe37c15ef70 2024-04-15T15:16:39.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.663+0000 7fe3727fc640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe36006c8f0 0x7fe36006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.663+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe37408b380 con 0x7fe37c15ef70 2024-04-15T15:16:39.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.663+0000 7fe381cce640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe36006c8f0 0x7fe36006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.664+0000 7fe381cce640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe36006c8f0 0x7fe36006edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe378038000 tx=0x7fe378008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.667+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe37c0cd3b0 con 0x7fe37c15ef70 2024-04-15T15:16:39.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:39.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:39.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.875+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe37c0689e0 con 0x7fe37c15ef70 2024-04-15T15:16:39.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.877+0000 7fe3727fc640 1 -- 172.21.15.46:0/1317302204 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7fe374058df0 con 0x7fe37c15ef70 2024-04-15T15:16:39.879 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:39.879 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe36006c8f0 msgr2=0x7fe36006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe36006c8f0 0x7fe36006edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe378038000 tx=0x7fe378008040 comp rx=0 tx=0).stop 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 msgr2=0x7fe37c155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 0x7fe37c155c90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe37400c780 tx=0x7fe37400cc40 comp rx=0 tx=0).stop 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 shutdown_connections 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe36006c8f0 0x7fe36006edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe37c15ef70 0x7fe37c155c90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe37c0d0690 0x7fe37c1594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 --2- 172.21.15.46:0/1317302204 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe37c0b3160 0x7fe37c14c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 >> 172.21.15.46:0/1317302204 conn(0x7fe37c06be00 msgr2=0x7fe37c06e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 shutdown_connections 2024-04-15T15:16:39.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:39.879+0000 7fe383f32640 1 -- 172.21.15.46:0/1317302204 wait complete. 2024-04-15T15:16:39.892 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:39.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.000+0000 7f33e0ba8640 1 Processor -- start 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.000+0000 7f33e0ba8640 1 -- start start 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.000+0000 7f33e0ba8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33dc0b49d0 0x7f33dc160af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.000+0000 7f33e0ba8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33dc09b020 0x7f33dc0ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.000+0000 7f33e0ba8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33dc0d14b0 con 0x7f33dc09ac40 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.000+0000 7f33e0ba8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f33dc0b1b70 con 0x7f33dc09b020 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33e0ba8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f33dc0cd550 con 0x7f33dc0b49d0 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33dc09b020 0x7f33dc0ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f33dc09b020 0x7f33dc0ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41310/0 (socket says 172.21.15.46:41310) 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 -- 172.21.15.46:0/580191931 learned_addr learned my addr 172.21.15.46:0/580191931 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 -- 172.21.15.46:0/580191931 >> v2:172.21.15.78:3300/0 conn(0x7f33dc0b49d0 msgr2=0x7f33dc160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33d3fff640 1 --2- 172.21.15.46:0/580191931 >> v2:172.21.15.78:3300/0 conn(0x7f33dc0b49d0 0x7f33dc160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 --2- 172.21.15.46:0/580191931 >> v2:172.21.15.78:3300/0 conn(0x7f33dc0b49d0 0x7f33dc160af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 -- 172.21.15.46:0/580191931 >> v1:172.21.15.78:6789/0 conn(0x7f33dc09ac40 legacy=0x7f33dc0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33dad9d640 1 -- 172.21.15.46:0/580191931 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc0cd830 con 0x7f33dc09b020 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.001+0000 7f33d3fff640 1 --2- 172.21.15.46:0/580191931 >> v2:172.21.15.78:3300/0 conn(0x7f33dc0b49d0 0x7f33dc160af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33dad9d640 1 --2- 172.21.15.46:0/580191931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09b020 0x7f33dc0ce030 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0b8470 tx=0x7f33d402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e067b5b5f23b1d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33d9d9b640 1 -- 172.21.15.46:0/580191931 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d402cb20 con 0x7f33dc09b020 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33d9d9b640 1 -- 172.21.15.46:0/580191931 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d402ccc0 con 0x7f33dc09b020 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33d9d9b640 1 -- 172.21.15.46:0/580191931 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d4034610 con 0x7f33dc09b020 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33e0ba8640 1 -- 172.21.15.46:0/580191931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09b020 msgr2=0x7f33dc0ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/580191931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09b020 0x7f33dc0ce030 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0b8470 tx=0x7f33d402bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33e0ba8640 1 -- 172.21.15.46:0/580191931 shutdown_connections 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/580191931 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09b020 0x7f33dc0ce030 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.002+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/580191931 >> v2:172.21.15.78:3300/0 conn(0x7f33dc0b49d0 0x7f33dc160af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.003+0000 7f33e0ba8640 1 -- 172.21.15.46:0/580191931 >> 172.21.15.46:0/580191931 conn(0x7f33dc0b3eb0 msgr2=0x7f33dc0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.003+0000 7f33e0ba8640 1 -- 172.21.15.46:0/580191931 shutdown_connections 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.003+0000 7f33e0ba8640 1 -- 172.21.15.46:0/580191931 wait complete. 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.004+0000 7f33e0ba8640 1 Processor -- start 2024-04-15T15:16:40.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.004+0000 7f33e0ba8640 1 -- start start 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 0x7f33dc158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33dc09b020 0x7f33dc06caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33dc0b49d0 0x7f33dc06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc0ce570 con 0x7f33dc09ac40 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc161030 con 0x7f33dc09b020 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f33dc0ce6f0 con 0x7f33dc0b49d0 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 0x7f33dc158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 0x7f33dc158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41314/0 (socket says 172.21.15.46:41314) 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 -- 172.21.15.46:0/959931307 learned_addr learned my addr 172.21.15.46:0/959931307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33d3fff640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33dc09b020 0x7f33dc06caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33dad9d640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33dc0b49d0 0x7f33dc06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 -- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33dc0b49d0 msgr2=0x7f33dc06d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33dc0b49d0 0x7f33dc06d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 -- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33dc09b020 msgr2=0x7f33dc06caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33dc09b020 0x7f33dc06caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 -- 172.21.15.46:0/959931307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc0b5400 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33da59c640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 0x7f33dc158710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b3f0 tx=0x7f33cc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc015020 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33cc011830 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.005+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc0b56e0 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.006+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc011c20 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.006+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc0b5ca0 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.007+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc0689e0 con 0x7f33dc09ac40 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.007+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33cc018020 con 0x7f33dc09ac40 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.008+0000 7f33d2ffd640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33bc06c8f0 0x7f33bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.008+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f33cc08baa0 con 0x7f33dc09ac40 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.008+0000 7f33d3fff640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33bc06c8f0 0x7f33bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.008+0000 7f33d3fff640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33bc06c8f0 0x7f33bc06edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0b8070 tx=0x7f33c4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.012+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33cc01f030 con 0x7f33dc09ac40 2024-04-15T15:16:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.197+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f33dc067b80 con 0x7f33dc09ac40 2024-04-15T15:16:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.197+0000 7f33d2ffd640 1 -- 172.21.15.46:0/959931307 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f33cc059480 con 0x7f33dc09ac40 2024-04-15T15:16:40.199 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33bc06c8f0 msgr2=0x7f33bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33bc06c8f0 0x7f33bc06edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f33dc0b8070 tx=0x7f33c4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 msgr2=0x7f33dc158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 0x7f33dc158710 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f33cc00b3f0 tx=0x7f33cc00b8b0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 shutdown_connections 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33bc06c8f0 0x7f33bc06edb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33dc0b49d0 0x7f33dc06d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33dc09b020 0x7f33dc06caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 --2- 172.21.15.46:0/959931307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33dc09ac40 0x7f33dc158710 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 >> 172.21.15.46:0/959931307 conn(0x7f33dc0b3eb0 msgr2=0x7f33dc0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 shutdown_connections 2024-04-15T15:16:40.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.199+0000 7f33e0ba8640 1 -- 172.21.15.46:0/959931307 wait complete. 2024-04-15T15:16:40.212 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.318+0000 7f8b20081640 1 Processor -- start 2024-04-15T15:16:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.318+0000 7f8b20081640 1 -- start start 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b20081640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 0x7f8b1809adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b20081640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b180ced60 con 0x7f8b1809b780 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b20081640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b1809b2f0 con 0x7f8b1815ebb0 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b20081640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8b180cd8c0 con 0x7f8b1809a9d0 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b1e61e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8b1809b780 0x7f8b180b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40370/0 (socket says 172.21.15.46:40370) 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b1e61e640 1 -- 172.21.15.46:0/407002211 learned_addr learned my addr 172.21.15.46:0/407002211 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.319+0000 7f8b1de1d640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 0x7f8b1809adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3930234116 0 0) 0x7f8b10001490 con 0x7f8b1809b780 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b04003520 con 0x7f8b1809b780 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4224978262 0 0) 0x7f8b04003520 con 0x7f8b1809b780 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 msgr2=0x7f8b1809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 0x7f8b1809adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 >> v1:172.21.15.78:6789/0 conn(0x7f8b1815ebb0 legacy=0x7f8b1806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b180cccb0 con 0x7f8b1809b780 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f8b10001bc0 con 0x7f8b1809b780 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 >> v1:172.21.15.46:6789/0 conn(0x7f8b1809b780 legacy=0x7f8b180b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 0x7f8b040039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b04004d40 0x7f8b040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b180cd8c0 con 0x7f8b04003f20 2024-04-15T15:16:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0400ba20 con 0x7f8b04004d40 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8b0400bc20 con 0x7f8b1809a9d0 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1de1d640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 0x7f8b040039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1e61e640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b04004d40 0x7f8b040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1d61c640 1 -- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 msgr2=0x7f8b040039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b1809a9d0 0x7f8b040039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1d61c640 1 -- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b04004d40 msgr2=0x7f8b040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b04004d40 0x7f8b040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.320+0000 7f8b1d61c640 1 -- 172.21.15.46:0/407002211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b04003520 con 0x7f8b04003f20 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.321+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b04004800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8b08001860 tx=0x7f8b04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11ef32509620129 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.321+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08001430 con 0x7f8b04003f20 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.321+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b08032620 con 0x7f8b04003f20 2024-04-15T15:16:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.321+0000 7f8b1ce1b640 1 -- 172.21.15.46:0/407002211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08032a10 con 0x7f8b04003f20 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b20081640 1 -- 172.21.15.46:0/407002211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 msgr2=0x7f8b04004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b20081640 1 --2- 172.21.15.46:0/407002211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b04004800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8b08001860 tx=0x7f8b04005830 comp rx=0 tx=0).stop 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b1e61e640 1 -- 172.21.15.46:0/407002211 reap_dead start 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b20081640 1 -- 172.21.15.46:0/407002211 shutdown_connections 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b20081640 1 -- 172.21.15.46:0/407002211 >> 172.21.15.46:0/407002211 conn(0x7f8b180b3fb0 msgr2=0x7f8b180b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b20081640 1 -- 172.21.15.46:0/407002211 shutdown_connections 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.322+0000 7f8b20081640 1 -- 172.21.15.46:0/407002211 wait complete. 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.323+0000 7f8b20081640 1 Processor -- start 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.323+0000 7f8b20081640 1 -- start start 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b20081640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b180c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b20081640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b18161b80 0x7f8b180c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b20081640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b180ca4a0 0x7f8b181515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b20081640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b180cd550 con 0x7f8b04003f20 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b20081640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8b180cd270 con 0x7f8b180ca4a0 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b20081640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8b180b5310 con 0x7f8b18161b80 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b180c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b180c6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41322/0 (socket says 172.21.15.46:41322) 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 -- 172.21.15.46:0/2717716269 learned_addr learned my addr 172.21.15.46:0/2717716269 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b18161b80 0x7f8b180c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 -- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b18161b80 msgr2=0x7f8b180c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b18161b80 0x7f8b180c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 -- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b180ca4a0 msgr2=0x7f8b181515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1e61e640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b180ca4a0 0x7f8b181515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b180ca4a0 0x7f8b181515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1de1d640 1 -- 172.21.15.46:0/2717716269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1815efa0 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b18161b80 0x7f8b180c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.324+0000 7f8b1e61e640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b180ca4a0 0x7f8b181515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.325+0000 7f8b1de1d640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b180c6a00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8b140119a0 tx=0x7f8b14011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.325+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400faf0 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.325+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1400fc90 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.325+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b180c7350 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.325+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14014440 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.325+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1815f260 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.326+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b180690f0 con 0x7f8b04003f20 2024-04-15T15:16:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.326+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b14012040 con 0x7f8b04003f20 2024-04-15T15:16:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.327+0000 7f8b0e7fc640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8afc06c920 0x7f8afc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.327+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8b1408c5b0 con 0x7f8b04003f20 2024-04-15T15:16:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.327+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8afc06c920 0x7f8afc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.328+0000 7f8b1d61c640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8afc06c920 0x7f8afc06ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8b08035000 tx=0x7f8b08030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.331+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b14019030 con 0x7f8b04003f20 2024-04-15T15:16:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.518+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b180717d0 con 0x7f8b04003f20 2024-04-15T15:16:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.518+0000 7f8b0e7fc640 1 -- 172.21.15.46:0/2717716269 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f8b1405a670 con 0x7f8b04003f20 2024-04-15T15:16:40.520 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8afc06c920 msgr2=0x7f8afc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8afc06c920 0x7f8afc06ede0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8b08035000 tx=0x7f8b08030000 comp rx=0 tx=0).stop 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 msgr2=0x7f8b180c6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b180c6a00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8b140119a0 tx=0x7f8b14011e60 comp rx=0 tx=0).stop 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 shutdown_connections 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8afc06c920 0x7f8afc06ede0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b180ca4a0 0x7f8b181515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8b18161b80 0x7f8b180c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 --2- 172.21.15.46:0/2717716269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b04003f20 0x7f8b180c6a00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 >> 172.21.15.46:0/2717716269 conn(0x7f8b180b3fb0 msgr2=0x7f8b18070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 shutdown_connections 2024-04-15T15:16:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.520+0000 7f8b20081640 1 -- 172.21.15.46:0/2717716269 wait complete. 2024-04-15T15:16:40.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:40.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.639+0000 7facff488640 1 Processor -- start 2024-04-15T15:16:40.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.639+0000 7facff488640 1 -- start start 2024-04-15T15:16:40.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.640+0000 7facff488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.640+0000 7facff488640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 0x7facf815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.640+0000 7facff488640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facf80d0fb0 con 0x7facf815ef70 2024-04-15T15:16:40.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.640+0000 7facff488640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7facf815cb50 con 0x7facf80b3160 2024-04-15T15:16:40.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.640+0000 7facff488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7facf815bfd0 con 0x7facf80d0690 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfd224640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfca23640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 0x7facf815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfda25640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7facf815ef70 0x7facf815f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:51278/0 (socket says 172.21.15.46:51278) 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfda25640 1 -- 172.21.15.46:0/3419747146 learned_addr learned my addr 172.21.15.46:0/3419747146 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfca23640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 0x7facf815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:59668/0 (socket says 172.21.15.46:59668) 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfd224640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf80d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:58054/0 (socket says 172.21.15.46:58054) 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfd224640 1 -- 172.21.15.46:0/3419747146 >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 msgr2=0x7facf815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfd224640 1 --2- 172.21.15.46:0/3419747146 >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 0x7facf815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfd224640 1 -- 172.21.15.46:0/3419747146 >> v1:172.21.15.78:6789/0 conn(0x7facf815ef70 legacy=0x7facf815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfd224640 1 -- 172.21.15.46:0/3419747146 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf815c2b0 con 0x7facf80d0690 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.641+0000 7facfca23640 1 --2- 172.21.15.46:0/3419747146 >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 0x7facf815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.642+0000 7facfd224640 1 --2- 172.21.15.46:0/3419747146 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf80d0a70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7facf00097a0 tx=0x7facf002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2b23aaddb8fcb61 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.642+0000 7facf7fff640 1 -- 172.21.15.46:0/3419747146 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0035020 con 0x7facf80d0690 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.642+0000 7facf7fff640 1 -- 172.21.15.46:0/3419747146 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facf0031830 con 0x7facf80d0690 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.642+0000 7facf7fff640 1 -- 172.21.15.46:0/3419747146 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0031c00 con 0x7facf80d0690 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 -- 172.21.15.46:0/3419747146 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 msgr2=0x7facf80d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 --2- 172.21.15.46:0/3419747146 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf80d0a70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7facf00097a0 tx=0x7facf002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 -- 172.21.15.46:0/3419747146 shutdown_connections 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 --2- 172.21.15.46:0/3419747146 >> v2:172.21.15.78:3300/0 conn(0x7facf80b3160 0x7facf815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 --2- 172.21.15.46:0/3419747146 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf80d0a70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 -- 172.21.15.46:0/3419747146 >> 172.21.15.46:0/3419747146 conn(0x7facf806be00 msgr2=0x7facf806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 -- 172.21.15.46:0/3419747146 shutdown_connections 2024-04-15T15:16:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.643+0000 7facff488640 1 -- 172.21.15.46:0/3419747146 wait complete. 2024-04-15T15:16:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.644+0000 7facff488640 1 Processor -- start 2024-04-15T15:16:40.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.644+0000 7facff488640 1 -- start start 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facff488640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7facf80b3160 0x7facf80cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facff488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf81e4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facff488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 0x7facf80ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facff488640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facf80b4fb0 con 0x7facf815ef70 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facff488640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7facf815c870 con 0x7facf80b3160 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facff488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7facf815bbd0 con 0x7facf80d0690 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfda25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 0x7facf80ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfda25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 0x7facf80ae990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41324/0 (socket says 172.21.15.46:41324) 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfca23640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf81e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfda25640 1 -- 172.21.15.46:0/2124269740 learned_addr learned my addr 172.21.15.46:0/2124269740 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfd224640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7facf80b3160 0x7facf80cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfca23640 1 -- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7facf80b3160 msgr2=0x7facf80cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfca23640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7facf80b3160 0x7facf80cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfca23640 1 -- 172.21.15.46:0/2124269740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 msgr2=0x7facf80ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfca23640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 0x7facf80ae990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfca23640 1 -- 172.21.15.46:0/2124269740 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf80ce510 con 0x7facf80d0690 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.645+0000 7facfda25640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 0x7facf80ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.646+0000 7facfca23640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf81e4850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7facec010df0 tx=0x7facec00e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.646+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec01e070 con 0x7facf80d0690 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.646+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec011860 con 0x7facf80d0690 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.646+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf80ce7f0 con 0x7facf80d0690 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.646+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec011c50 con 0x7facf80d0690 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.646+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7facf80cedb0 con 0x7facf80d0690 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.647+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf80689e0 con 0x7facf80d0690 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.647+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7facec021070 con 0x7facf80d0690 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.648+0000 7facf5ffb640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7face406c8f0 0x7face406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.648+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7facec08e8f0 con 0x7facf80d0690 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.648+0000 7facfd224640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7face406c8f0 0x7face406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.649+0000 7facfd224640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7face406c8f0 0x7face406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7facf0036000 tx=0x7facf003c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.653+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7facec05c240 con 0x7facf80d0690 2024-04-15T15:16:40.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.839+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7facf80739e0 con 0x7facf80d0690 2024-04-15T15:16:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.839+0000 7facf5ffb640 1 -- 172.21.15.46:0/2124269740 <== mon.2 v2:172.21.15.78:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7facec060540 con 0x7facf80d0690 2024-04-15T15:16:40.841 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7face406c8f0 msgr2=0x7face406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7face406c8f0 0x7face406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7facf0036000 tx=0x7facf003c040 comp rx=0 tx=0).stop 2024-04-15T15:16:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 msgr2=0x7facf81e4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf81e4850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7facec010df0 tx=0x7facec00e730 comp rx=0 tx=0).stop 2024-04-15T15:16:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 shutdown_connections 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7face406c8f0 0x7face406edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facf815ef70 0x7facf80ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7facf80d0690 0x7facf81e4850 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 --2- 172.21.15.46:0/2124269740 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7facf80b3160 0x7facf80cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 >> 172.21.15.46:0/2124269740 conn(0x7facf806be00 msgr2=0x7facf8071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 shutdown_connections 2024-04-15T15:16:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.841+0000 7facff488640 1 -- 172.21.15.46:0/2124269740 wait complete. 2024-04-15T15:16:40.854 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.960+0000 7f69a7bc8640 1 Processor -- start 2024-04-15T15:16:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.960+0000 7f69a7bc8640 1 -- start start 2024-04-15T15:16:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a7bc8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a015e790 0x7f69a00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a7bc8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69a00b7be0 0x7f69a00d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a7bc8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f69a00d1b20 0x7f69a00d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a7bc8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f69a00d3790 con 0x7f69a00b7be0 2024-04-15T15:16:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a7bc8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f69a00d45b0 con 0x7f69a00d1b20 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a7bc8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f69a015eb70 con 0x7f69a015e790 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69a00b7be0 0x7f69a00d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69a00b7be0 0x7f69a00d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41326/0 (socket says 172.21.15.46:41326) 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 -- 172.21.15.46:0/2008073153 learned_addr learned my addr 172.21.15.46:0/2008073153 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5964640 1 --2- 172.21.15.46:0/2008073153 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a015e790 0x7f69a00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a6165640 1 --2- 172.21.15.46:0/2008073153 >> v2:172.21.15.78:3300/0 conn(0x7f69a00d1b20 0x7f69a00d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 -- 172.21.15.46:0/2008073153 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a015e790 msgr2=0x7f69a00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 --2- 172.21.15.46:0/2008073153 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a015e790 0x7f69a00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 -- 172.21.15.46:0/2008073153 >> v2:172.21.15.78:3300/0 conn(0x7f69a00d1b20 msgr2=0x7f69a00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 --2- 172.21.15.46:0/2008073153 >> v2:172.21.15.78:3300/0 conn(0x7f69a00d1b20 0x7f69a00d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.961+0000 7f69a5163640 1 -- 172.21.15.46:0/2008073153 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a00d3c80 con 0x7f69a00b7be0 2024-04-15T15:16:40.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.962+0000 7f69a5163640 1 --2- 172.21.15.46:0/2008073153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a00b7be0 0x7f69a00d15e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69a00cebe0 tx=0x7f699002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576fc3e4778dedd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.962+0000 7f69a4962640 1 -- 172.21.15.46:0/2008073153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990038070 con 0x7f69a00b7be0 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.962+0000 7f69a4962640 1 -- 172.21.15.46:0/2008073153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699002f050 con 0x7f69a00b7be0 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.962+0000 7f69a4962640 1 -- 172.21.15.46:0/2008073153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990031970 con 0x7f69a00b7be0 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 -- 172.21.15.46:0/2008073153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a00b7be0 msgr2=0x7f69a00d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/2008073153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a00b7be0 0x7f69a00d15e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f69a00cebe0 tx=0x7f699002b7f0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 -- 172.21.15.46:0/2008073153 shutdown_connections 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/2008073153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a00b7be0 0x7f69a00d15e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/2008073153 >> v2:172.21.15.78:3300/0 conn(0x7f69a00d1b20 0x7f69a00d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/2008073153 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a015e790 0x7f69a00b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 -- 172.21.15.46:0/2008073153 >> 172.21.15.46:0/2008073153 conn(0x7f69a0067b40 msgr2=0x7f69a0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 -- 172.21.15.46:0/2008073153 shutdown_connections 2024-04-15T15:16:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.963+0000 7f69a7bc8640 1 -- 172.21.15.46:0/2008073153 wait complete. 2024-04-15T15:16:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.964+0000 7f69a7bc8640 1 Processor -- start 2024-04-15T15:16:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.964+0000 7f69a7bc8640 1 -- start start 2024-04-15T15:16:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a7bc8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00b7be0 0x7f69a00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a7bc8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a00d1b20 0x7f69a01f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a7bc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 0x7f69a00ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a7bc8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69a00d4f80 con 0x7f69a015e790 2024-04-15T15:16:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a7bc8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f69a00d4240 con 0x7f69a00b7be0 2024-04-15T15:16:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a7bc8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f69a00d5100 con 0x7f69a00d1b20 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 0x7f69a00ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 0x7f69a00ab190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41338/0 (socket says 172.21.15.46:41338) 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 -- 172.21.15.46:0/1019032532 learned_addr learned my addr 172.21.15.46:0/1019032532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a5163640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a00d1b20 0x7f69a01f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a5964640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00b7be0 0x7f69a00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 -- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a00d1b20 msgr2=0x7f69a01f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a00d1b20 0x7f69a01f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 -- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00b7be0 msgr2=0x7f69a00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00b7be0 0x7f69a00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 -- 172.21.15.46:0/1019032532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a01f49e0 con 0x7f69a015e790 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69a6165640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 0x7f69a00ab190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f699800e860 tx=0x7f699800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.965+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6998017070 con 0x7f69a015e790 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.966+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699800c6d0 con 0x7f69a015e790 2024-04-15T15:16:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.966+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a00ab6d0 con 0x7f69a015e790 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.966+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699800cac0 con 0x7f69a015e790 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.966+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a0160110 con 0x7f69a015e790 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.967+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a00689e0 con 0x7f69a015e790 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.967+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f699801a070 con 0x7f69a015e790 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.968+0000 7f69967fc640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f698406c920 0x7f698406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.968+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f699808bf70 con 0x7f69a015e790 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.968+0000 7f69a5964640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f698406c920 0x7f698406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.968+0000 7f69a5964640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f698406c920 0x7f698406ede0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f699c011340 tx=0x7f699c00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:40.972+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69980598b0 con 0x7f69a015e790 2024-04-15T15:16:41.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.155+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f69a00b6c30 con 0x7f69a015e790 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.156+0000 7f69967fc640 1 -- 172.21.15.46:0/1019032532 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f69980600d0 con 0x7f69a015e790 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f698406c920 msgr2=0x7f698406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f698406c920 0x7f698406ede0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f699c011340 tx=0x7f699c00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 msgr2=0x7f69a00ab190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 0x7f69a00ab190 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f699800e860 tx=0x7f699800ed20 comp rx=0 tx=0).stop 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 shutdown_connections 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f698406c920 0x7f698406ede0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69a015e790 0x7f69a00ab190 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f69a00d1b20 0x7f69a01f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 --2- 172.21.15.46:0/1019032532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f69a00b7be0 0x7f69a00b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 >> 172.21.15.46:0/1019032532 conn(0x7f69a0067b40 msgr2=0x7f69a00d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 shutdown_connections 2024-04-15T15:16:41.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.158+0000 7f69a7bc8640 1 -- 172.21.15.46:0/1019032532 wait complete. 2024-04-15T15:16:41.171 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.274+0000 7f75bbd3d640 1 Processor -- start 2024-04-15T15:16:41.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.275+0000 7f75bbd3d640 1 -- start start 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.275+0000 7f75bbd3d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75b415e790 0x7f75b40b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.275+0000 7f75bbd3d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40b7be0 0x7f75b40d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.275+0000 7f75bbd3d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b415eb70 con 0x7f75b40d1b20 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.275+0000 7f75bbd3d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f75b40d3790 con 0x7f75b415e790 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.275+0000 7f75bbd3d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f75b40d45b0 con 0x7f75b40b7be0 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75b92d8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40b7be0 0x7f75b40d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75b9ad9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75b415e790 0x7f75b40b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75ba2da640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f75b40d1b20 0x7f75b40d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:51288/0 (socket says 172.21.15.46:51288) 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75ba2da640 1 -- 172.21.15.46:0/3032571098 learned_addr learned my addr 172.21.15.46:0/3032571098 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:41.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75b92d8640 1 -- 172.21.15.46:0/3032571098 >> v2:172.21.15.78:3300/0 conn(0x7f75b415e790 msgr2=0x7f75b40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75b92d8640 1 --2- 172.21.15.46:0/3032571098 >> v2:172.21.15.78:3300/0 conn(0x7f75b415e790 0x7f75b40b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75b92d8640 1 -- 172.21.15.46:0/3032571098 >> v1:172.21.15.78:6789/0 conn(0x7f75b40d1b20 legacy=0x7f75b40d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.276+0000 7f75b92d8640 1 -- 172.21.15.46:0/3032571098 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b40d3c80 con 0x7f75b40b7be0 2024-04-15T15:16:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75b92d8640 1 --2- 172.21.15.46:0/3032571098 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40b7be0 0x7f75b40d15e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75a40097a0 tx=0x7f75a402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=780eca0545a8f97c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75b8ad7640 1 -- 172.21.15.46:0/3032571098 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4038070 con 0x7f75b40b7be0 2024-04-15T15:16:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75b8ad7640 1 -- 172.21.15.46:0/3032571098 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a4031700 con 0x7f75b40b7be0 2024-04-15T15:16:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75b8ad7640 1 -- 172.21.15.46:0/3032571098 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4031ad0 con 0x7f75b40b7be0 2024-04-15T15:16:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75bbd3d640 1 -- 172.21.15.46:0/3032571098 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40b7be0 msgr2=0x7f75b40d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/3032571098 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40b7be0 0x7f75b40d15e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f75a40097a0 tx=0x7f75a402b7a0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75bbd3d640 1 -- 172.21.15.46:0/3032571098 shutdown_connections 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/3032571098 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40b7be0 0x7f75b40d15e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/3032571098 >> v2:172.21.15.78:3300/0 conn(0x7f75b415e790 0x7f75b40b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.277+0000 7f75bbd3d640 1 -- 172.21.15.46:0/3032571098 >> 172.21.15.46:0/3032571098 conn(0x7f75b4067b40 msgr2=0x7f75b4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.278+0000 7f75bbd3d640 1 -- 172.21.15.46:0/3032571098 shutdown_connections 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.278+0000 7f75bbd3d640 1 -- 172.21.15.46:0/3032571098 wait complete. 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 Processor -- start 2024-04-15T15:16:41.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 -- start start 2024-04-15T15:16:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 0x7f75b414fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40d1b20 0x7f75b40ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75b415e790 0x7f75b40b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75b40d5150 con 0x7f75b40b7be0 2024-04-15T15:16:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75b40d4240 con 0x7f75b415e790 2024-04-15T15:16:41.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.279+0000 7f75bbd3d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f75b40d52d0 con 0x7f75b40d1b20 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 0x7f75b414fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 0x7f75b414fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41350/0 (socket says 172.21.15.46:41350) 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b92d8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40d1b20 0x7f75b40ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 -- 172.21.15.46:0/59232900 learned_addr learned my addr 172.21.15.46:0/59232900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75ba2da640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75b415e790 0x7f75b40b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 -- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40d1b20 msgr2=0x7f75b40ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40d1b20 0x7f75b40ab960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 -- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75b415e790 msgr2=0x7f75b40b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75b415e790 0x7f75b40b8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 -- 172.21.15.46:0/59232900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b414ffa0 con 0x7f75b40b7be0 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b92d8640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40d1b20 0x7f75b40ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75b9ad9640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 0x7f75b414fa60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e8a0 tx=0x7f75b000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0017070 con 0x7f75b40b7be0 2024-04-15T15:16:41.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b000c940 con 0x7f75b40b7be0 2024-04-15T15:16:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b4151dd0 con 0x7f75b40b7be0 2024-04-15T15:16:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b000cd30 con 0x7f75b40b7be0 2024-04-15T15:16:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.280+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b4152350 con 0x7f75b40b7be0 2024-04-15T15:16:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.281+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b40689e0 con 0x7f75b40b7be0 2024-04-15T15:16:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.282+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75b001a070 con 0x7f75b40b7be0 2024-04-15T15:16:41.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.282+0000 7f75aa7fc640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f759806c8f0 0x7f759806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.282+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f75b008bd10 con 0x7f75b40b7be0 2024-04-15T15:16:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.282+0000 7f75b92d8640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f759806c8f0 0x7f759806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.283+0000 7f75b92d8640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f759806c8f0 0x7f759806edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f75a4035000 tx=0x7f75a4008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.286+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75b414ffa0 con 0x7f75b40b7be0 2024-04-15T15:16:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.485+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f75b40d17f0 con 0x7f75b40b7be0 2024-04-15T15:16:41.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.486+0000 7f75aa7fc640 1 -- 172.21.15.46:0/59232900 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f75b0059700 con 0x7f75b40b7be0 2024-04-15T15:16:41.488 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f759806c8f0 msgr2=0x7f759806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f759806c8f0 0x7f759806edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f75a4035000 tx=0x7f75a4008030 comp rx=0 tx=0).stop 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 msgr2=0x7f75b414fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 0x7f75b414fa60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e8a0 tx=0x7f75b000ed60 comp rx=0 tx=0).stop 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 shutdown_connections 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f759806c8f0 0x7f759806edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75b415e790 0x7f75b40b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75b40d1b20 0x7f75b40ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 --2- 172.21.15.46:0/59232900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75b40b7be0 0x7f75b414fa60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 >> 172.21.15.46:0/59232900 conn(0x7f75b4067b40 msgr2=0x7f75b40d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 shutdown_connections 2024-04-15T15:16:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.488+0000 7f75bbd3d640 1 -- 172.21.15.46:0/59232900 wait complete. 2024-04-15T15:16:41.501 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-04-15T15:16:41.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:41.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:41.530 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:41.530 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:16:41.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:41.530 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:16:41.585 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:16:41.586 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:41.603 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:16:41.603 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:41.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.762+0000 7f43941bd640 1 Processor -- start 2024-04-15T15:16:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.762+0000 7f43941bd640 1 -- start start 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f43941bd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 0x7f438c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f43941bd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f438c071d90 0x7f438c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f43941bd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f438c06ff30 con 0x7f438c071240 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f43941bd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f438c070820 con 0x7f438c071d90 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f43941bd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f438c0cecc0 con 0x7f438c0ce8e0 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f439275a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f438c071240 0x7f438c160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40372/0 (socket says 172.21.15.46:40372) 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f439275a640 1 -- 172.21.15.46:0/2421316602 learned_addr learned my addr 172.21.15.46:0/2421316602 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f4391f59640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 0x7f438c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869218820 0 0) 0x7f438c06ff30 con 0x7f438c071240 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f4391758640 1 --2- 172.21.15.46:0/2421316602 >> v2:172.21.15.78:3300/0 conn(0x7f438c071d90 0x7f438c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.763+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4374003520 con 0x7f438c071240 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 277067117 0 0) 0x7f4374003520 con 0x7f438c071240 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 msgr2=0x7f438c070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 0x7f438c070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 >> v2:172.21.15.78:3300/0 conn(0x7f438c071d90 msgr2=0x7f438c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 --2- 172.21.15.46:0/2421316602 >> v2:172.21.15.78:3300/0 conn(0x7f438c071d90 0x7f438c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438c09b9d0 con 0x7f438c071240 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f4384001690 con 0x7f438c071240 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 >> v1:172.21.15.46:6789/0 conn(0x7f438c071240 legacy=0x7f438c160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43740039b0 0x7f4374003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 0x7f4374004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4374004d40 0x7f43740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f438c070820 con 0x7f4374004d40 2024-04-15T15:16:41.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f438c0cecc0 con 0x7f43740039b0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f437400b9c0 con 0x7f438c0ce8e0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391f59640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43740039b0 0x7f4374003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391758640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 0x7f4374004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f439275a640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4374004d40 0x7f43740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391f59640 1 -- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 msgr2=0x7f4374004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391f59640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c0ce8e0 0x7f4374004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391f59640 1 -- 172.21.15.46:0/2421316602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4374004d40 msgr2=0x7f43740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391f59640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4374004d40 0x7f43740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.764+0000 7f4391f59640 1 -- 172.21.15.46:0/2421316602 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4374003520 con 0x7f43740039b0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.765+0000 7f4391f59640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43740039b0 0x7f4374003dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f437c009fc0 tx=0x7f4374005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=826ab2cf4c59ee27 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.765+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c039070 con 0x7f43740039b0 2024-04-15T15:16:41.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.765+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437c02cc20 con 0x7f43740039b0 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.765+0000 7f4390f57640 1 -- 172.21.15.46:0/2421316602 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c009c10 con 0x7f43740039b0 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f43941bd640 1 -- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43740039b0 msgr2=0x7f4374003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f43941bd640 1 --2- 172.21.15.46:0/2421316602 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f43740039b0 0x7f4374003dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f437c009fc0 tx=0x7f4374005830 comp rx=0 tx=0).stop 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f439275a640 1 -- 172.21.15.46:0/2421316602 reap_dead start 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f43941bd640 1 -- 172.21.15.46:0/2421316602 shutdown_connections 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f43941bd640 1 -- 172.21.15.46:0/2421316602 >> 172.21.15.46:0/2421316602 conn(0x7f438c0b3f40 msgr2=0x7f438c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f43941bd640 1 -- 172.21.15.46:0/2421316602 shutdown_connections 2024-04-15T15:16:41.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.766+0000 7f43941bd640 1 -- 172.21.15.46:0/2421316602 wait complete. 2024-04-15T15:16:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.767+0000 7f43941bd640 1 Processor -- start 2024-04-15T15:16:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.767+0000 7f43941bd640 1 -- start start 2024-04-15T15:16:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 0x7f438c152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c1f6640 0x7f438c1f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f438c14c4d0 0x7f438c14c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f438c0704b0 con 0x7f43740039b0 2024-04-15T15:16:41.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f438c0701d0 con 0x7f438c14c4d0 2024-04-15T15:16:41.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f438c071970 con 0x7f438c1f6640 2024-04-15T15:16:41.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 0x7f438c152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 0x7f438c152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41368/0 (socket says 172.21.15.46:41368) 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 -- 172.21.15.46:0/3557867389 learned_addr learned my addr 172.21.15.46:0/3557867389 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391758640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c1f6640 0x7f438c1f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f439275a640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f438c14c4d0 0x7f438c14c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 -- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c1f6640 msgr2=0x7f438c1f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c1f6640 0x7f438c1f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 -- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f438c14c4d0 msgr2=0x7f438c14c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f438c14c4d0 0x7f438c14c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 -- 172.21.15.46:0/3557867389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438c0b5e50 con 0x7f43740039b0 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f4391f59640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 0x7f438c152490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f437c037040 tx=0x7f437c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c039050 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f437c03a040 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f438c0b6130 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c034e60 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.768+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f438c0b66f0 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.770+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f438c0689e0 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.770+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f437c047030 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.770+0000 7f438a7fc640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f437806c8f0 0x7f437806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.770+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f437c0ae1d0 con 0x7f43740039b0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.770+0000 7f4391758640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f437806c8f0 0x7f437806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.771+0000 7f4391758640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f437806c8f0 0x7f437806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f438c09d860 tx=0x7f4380009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.775+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f437c04e060 con 0x7f43740039b0 2024-04-15T15:16:41.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.959+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f438c0b4ca0 con 0x7f43740039b0 2024-04-15T15:16:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.960+0000 7f438a7fc640 1 -- 172.21.15.46:0/3557867389 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f437c07c350 con 0x7f43740039b0 2024-04-15T15:16:41.962 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:41.963 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f437806c8f0 msgr2=0x7f437806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f437806c8f0 0x7f437806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f438c09d860 tx=0x7f4380009190 comp rx=0 tx=0).stop 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 msgr2=0x7f438c152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 0x7f438c152490 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f437c037040 tx=0x7f437c009750 comp rx=0 tx=0).stop 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 shutdown_connections 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f437806c8f0 0x7f437806edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f438c14c4d0 0x7f438c14c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f438c1f6640 0x7f438c1f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 --2- 172.21.15.46:0/3557867389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43740039b0 0x7f438c152490 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 >> 172.21.15.46:0/3557867389 conn(0x7f438c0b3f40 msgr2=0x7f438c0b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 shutdown_connections 2024-04-15T15:16:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:41.962+0000 7f43941bd640 1 -- 172.21.15.46:0/3557867389 wait complete. 2024-04-15T15:16:41.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:41.976 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:42.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.081+0000 7f845e729640 1 Processor -- start 2024-04-15T15:16:42.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f845e729640 1 -- start start 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f845e729640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f845815ebb0 0x7f845806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f845e729640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84580ced60 con 0x7f845809a9d0 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f845e729640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84580cccb0 con 0x7f845809b780 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f845e729640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f84580cd8c0 con 0x7f845815ebb0 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f84577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f845815ebb0 0x7f845806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f8457fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f845809a9d0 0x7f845809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40384/0 (socket says 172.21.15.46:40384) 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f8457fff640 1 -- 172.21.15.46:0/2878231268 learned_addr learned my addr 172.21.15.46:0/2878231268 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.082+0000 7f84577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f845815ebb0 0x7f845806cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41378/0 (socket says 172.21.15.46:41378) 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.083+0000 7f84577fe640 1 -- 172.21.15.46:0/2878231268 >> v1:172.21.15.78:6789/0 conn(0x7f845809b780 legacy=0x7f84580b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.083+0000 7f84577fe640 1 -- 172.21.15.46:0/2878231268 >> v1:172.21.15.46:6789/0 conn(0x7f845809a9d0 legacy=0x7f845809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.083+0000 7f84577fe640 1 -- 172.21.15.46:0/2878231268 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84580ccf90 con 0x7f845815ebb0 2024-04-15T15:16:42.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.083+0000 7f8456ffd640 1 -- 172.21.15.46:0/2878231268 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983358721 0 0) 0x7f8448001110 con 0x7f845809a9d0 2024-04-15T15:16:42.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.083+0000 7f84577fe640 1 --2- 172.21.15.46:0/2878231268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845815ebb0 0x7f845806cd00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f844c0097a0 tx=0x7f844c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b5835e36b165f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f8456ffd640 1 -- 172.21.15.46:0/2878231268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c034070 con 0x7f845815ebb0 2024-04-15T15:16:42.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f8456ffd640 1 -- 172.21.15.46:0/2878231268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844c030610 con 0x7f845815ebb0 2024-04-15T15:16:42.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f8456ffd640 1 -- 172.21.15.46:0/2878231268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c0309e0 con 0x7f845815ebb0 2024-04-15T15:16:42.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 -- 172.21.15.46:0/2878231268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845815ebb0 msgr2=0x7f845806cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 --2- 172.21.15.46:0/2878231268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845815ebb0 0x7f845806cd00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f844c0097a0 tx=0x7f844c02b960 comp rx=0 tx=0).stop 2024-04-15T15:16:42.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 -- 172.21.15.46:0/2878231268 shutdown_connections 2024-04-15T15:16:42.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 --2- 172.21.15.46:0/2878231268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845815ebb0 0x7f845806cd00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 -- 172.21.15.46:0/2878231268 >> 172.21.15.46:0/2878231268 conn(0x7f84580b3fb0 msgr2=0x7f84580b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 -- 172.21.15.46:0/2878231268 shutdown_connections 2024-04-15T15:16:42.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.084+0000 7f845e729640 1 -- 172.21.15.46:0/2878231268 wait complete. 2024-04-15T15:16:42.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.085+0000 7f845e729640 1 Processor -- start 2024-04-15T15:16:42.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 -- start start 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845809a9d0 0x7f84580cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 0x7f84580c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f84580c9ad0 0x7f845815fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f845806d240 con 0x7f845809b780 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f84580cd550 con 0x7f845809a9d0 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f845e729640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f84580b5310 con 0x7f84580c9ad0 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f84577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 0x7f84580c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f84577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 0x7f84580c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41394/0 (socket says 172.21.15.46:41394) 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.086+0000 7f84577fe640 1 -- 172.21.15.46:0/161763719 learned_addr learned my addr 172.21.15.46:0/161763719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f8457fff640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845809a9d0 0x7f84580cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f845ccc6640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f84580c9ad0 0x7f845815fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f84577fe640 1 -- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f84580c9ad0 msgr2=0x7f845815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f84577fe640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f84580c9ad0 0x7f845815fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f84577fe640 1 -- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845809a9d0 msgr2=0x7f84580cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f84577fe640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845809a9d0 0x7f84580cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f84577fe640 1 -- 172.21.15.46:0/161763719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8458160c70 con 0x7f845809b780 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f845ccc6640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f84580c9ad0 0x7f845815fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f84577fe640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 0x7f84580c9590 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f844c000c00 tx=0x7f844c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c047070 con 0x7f845809b780 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844c004030 con 0x7f845809b780 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8458160f50 con 0x7f845809b780 2024-04-15T15:16:42.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844c034040 con 0x7f845809b780 2024-04-15T15:16:42.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.087+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8458161510 con 0x7f845809b780 2024-04-15T15:16:42.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.088+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84580c9ee0 con 0x7f845809b780 2024-04-15T15:16:42.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.089+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f844c04a070 con 0x7f845809b780 2024-04-15T15:16:42.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.089+0000 7f8454ff9640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f844406c920 0x7f844406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.089+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f844c0ad8e0 con 0x7f845809b780 2024-04-15T15:16:42.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.089+0000 7f8457fff640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f844406c920 0x7f844406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.090+0000 7f8457fff640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f844406c920 0x7f844406ede0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8448000f60 tx=0x7f8448007970 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.093+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84580c9ee0 con 0x7f845809b780 2024-04-15T15:16:42.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.275+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84580690f0 con 0x7f845809b780 2024-04-15T15:16:42.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.276+0000 7f8454ff9640 1 -- 172.21.15.46:0/161763719 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f844c07b910 con 0x7f845809b780 2024-04-15T15:16:42.278 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f844406c920 msgr2=0x7f844406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f844406c920 0x7f844406ede0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8448000f60 tx=0x7f8448007970 comp rx=0 tx=0).stop 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 msgr2=0x7f84580c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 0x7f84580c9590 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f844c000c00 tx=0x7f844c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 shutdown_connections 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f844406c920 0x7f844406ede0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f84580c9ad0 0x7f845815fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f845809b780 0x7f84580c9590 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 --2- 172.21.15.46:0/161763719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f845809a9d0 0x7f84580cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 >> 172.21.15.46:0/161763719 conn(0x7f84580b3fb0 msgr2=0x7f8458070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 shutdown_connections 2024-04-15T15:16:42.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.278+0000 7f845e729640 1 -- 172.21.15.46:0/161763719 wait complete. 2024-04-15T15:16:42.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.397+0000 7f86511a8640 1 Processor -- start 2024-04-15T15:16:42.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f86511a8640 1 -- start start 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f86511a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f864c15e830 0x7f864c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f86511a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f864c15ec10 0x7f864c067720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f86511a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f864c0b3c10 con 0x7f864c067c60 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f86511a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f864c0b3ef0 con 0x7f864c15ec10 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f86511a8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f864c0b41d0 con 0x7f864c15e830 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.398+0000 7f864a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f864c15ec10 0x7f864c067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f864c15ec10 0x7f864c067720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41404/0 (socket says 172.21.15.46:41404) 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864b59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f864c067c60 0x7f864c0b53c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40388/0 (socket says 172.21.15.46:40388) 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864a59c640 1 -- 172.21.15.46:0/670661043 learned_addr learned my addr 172.21.15.46:0/670661043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864ad9d640 1 --2- 172.21.15.46:0/670661043 >> v2:172.21.15.78:3300/0 conn(0x7f864c15e830 0x7f864c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864a59c640 1 -- 172.21.15.46:0/670661043 >> v2:172.21.15.78:3300/0 conn(0x7f864c15e830 msgr2=0x7f864c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864a59c640 1 --2- 172.21.15.46:0/670661043 >> v2:172.21.15.78:3300/0 conn(0x7f864c15e830 0x7f864c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864a59c640 1 -- 172.21.15.46:0/670661043 >> v1:172.21.15.46:6789/0 conn(0x7f864c067c60 legacy=0x7f864c0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864a59c640 1 -- 172.21.15.46:0/670661043 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c0b44b0 con 0x7f864c15ec10 2024-04-15T15:16:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f8649d9b640 1 -- 172.21.15.46:0/670661043 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176834019 0 0) 0x7f864c0b3c10 con 0x7f864c067c60 2024-04-15T15:16:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.399+0000 7f864ad9d640 1 --2- 172.21.15.46:0/670661043 >> v2:172.21.15.78:3300/0 conn(0x7f864c15e830 0x7f864c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f864a59c640 1 --2- 172.21.15.46:0/670661043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c15ec10 0x7f864c067720 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f864c073d70 tx=0x7f864002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1efa68b12406 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f8649d9b640 1 -- 172.21.15.46:0/670661043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640035020 con 0x7f864c15ec10 2024-04-15T15:16:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f8649d9b640 1 -- 172.21.15.46:0/670661043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8640031870 con 0x7f864c15ec10 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f8649d9b640 1 -- 172.21.15.46:0/670661043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640031c40 con 0x7f864c15ec10 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 -- 172.21.15.46:0/670661043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c15ec10 msgr2=0x7f864c067720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 --2- 172.21.15.46:0/670661043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c15ec10 0x7f864c067720 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f864c073d70 tx=0x7f864002b910 comp rx=0 tx=0).stop 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 -- 172.21.15.46:0/670661043 shutdown_connections 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 --2- 172.21.15.46:0/670661043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c15ec10 0x7f864c067720 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 --2- 172.21.15.46:0/670661043 >> v2:172.21.15.78:3300/0 conn(0x7f864c15e830 0x7f864c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 -- 172.21.15.46:0/670661043 >> 172.21.15.46:0/670661043 conn(0x7f864c073360 msgr2=0x7f864c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 -- 172.21.15.46:0/670661043 shutdown_connections 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.400+0000 7f86511a8640 1 -- 172.21.15.46:0/670661043 wait complete. 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.402+0000 7f86511a8640 1 Processor -- start 2024-04-15T15:16:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.402+0000 7f86511a8640 1 -- start start 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.402+0000 7f86511a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f864c067c60 0x7f864c156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f864c15e830 0x7f864c1423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 0x7f864c155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1600e0 con 0x7f864c155480 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f864c15ff60 con 0x7f864c15e830 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f864c0b5ad0 con 0x7f864c067c60 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 0x7f864c155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 0x7f864c155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41420/0 (socket says 172.21.15.46:41420) 2024-04-15T15:16:42.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 -- 172.21.15.46:0/1935306946 learned_addr learned my addr 172.21.15.46:0/1935306946 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864ad9d640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f864c067c60 0x7f864c156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864a59c640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f864c15e830 0x7f864c1423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 -- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f864c067c60 msgr2=0x7f864c156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f864c067c60 0x7f864c156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 -- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f864c15e830 msgr2=0x7f864c1423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f864c15e830 0x7f864c1423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 -- 172.21.15.46:0/1935306946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c1428e0 con 0x7f864c155480 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f864b59e640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 0x7f864c155910 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f864400c7a0 tx=0x7f864400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644018070 con 0x7f864c155480 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f864400a640 con 0x7f864c155480 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c155f40 con 0x7f864c155480 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864400aab0 con 0x7f864c155480 2024-04-15T15:16:42.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.403+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c156500 con 0x7f864c155480 2024-04-15T15:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.404+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864c0b6540 con 0x7f864c155480 2024-04-15T15:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.405+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f864401b070 con 0x7f864c155480 2024-04-15T15:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.405+0000 7f86337fe640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f863806c8f0 0x7f863806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.405+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f864408b4d0 con 0x7f864c155480 2024-04-15T15:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.406+0000 7f864ad9d640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f863806c8f0 0x7f863806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.406+0000 7f864ad9d640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f863806c8f0 0x7f863806edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.410+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f864c0b6540 con 0x7f864c155480 2024-04-15T15:16:42.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.595+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f864c0689e0 con 0x7f864c155480 2024-04-15T15:16:42.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.595+0000 7f86337fe640 1 -- 172.21.15.46:0/1935306946 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f8644058eb0 con 0x7f864c155480 2024-04-15T15:16:42.597 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f863806c8f0 msgr2=0x7f863806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f863806c8f0 0x7f863806edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f863c006fd0 tx=0x7f863c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 msgr2=0x7f864c155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 0x7f864c155910 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f864400c7a0 tx=0x7f864400cc60 comp rx=0 tx=0).stop 2024-04-15T15:16:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 shutdown_connections 2024-04-15T15:16:42.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f863806c8f0 0x7f863806edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f864c155480 0x7f864c155910 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f864c15e830 0x7f864c1423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 --2- 172.21.15.46:0/1935306946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f864c067c60 0x7f864c156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 >> 172.21.15.46:0/1935306946 conn(0x7f864c073360 msgr2=0x7f864c0b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 shutdown_connections 2024-04-15T15:16:42.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.597+0000 7f86511a8640 1 -- 172.21.15.46:0/1935306946 wait complete. 2024-04-15T15:16:42.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.718+0000 7f0f338a0640 1 Processor -- start 2024-04-15T15:16:42.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.718+0000 7f0f338a0640 1 -- start start 2024-04-15T15:16:42.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f338a0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f2c071d10 0x7f0f2c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f338a0640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f338a0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2c15db40 con 0x7f0f2c072d80 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f338a0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0f2c072170 con 0x7f0f2c071d10 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f338a0640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c15cfc0 con 0x7f0f2c0d0bd0 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f30e3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f2c071d10 0x7f0f2c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f3163c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f2c072d80 0x7f0f2c0af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40394/0 (socket says 172.21.15.46:40394) 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f30e3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f2c071d10 0x7f0f2c0d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41428/0 (socket says 172.21.15.46:41428) 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f3163c640 1 -- 172.21.15.46:0/1420683003 learned_addr learned my addr 172.21.15.46:0/1420683003 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f31e3d640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f30e3b640 1 -- 172.21.15.46:0/1420683003 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 msgr2=0x7f0f2c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f30e3b640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f30e3b640 1 -- 172.21.15.46:0/1420683003 >> v1:172.21.15.46:6789/0 conn(0x7f0f2c072d80 legacy=0x7f0f2c0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f30e3b640 1 -- 172.21.15.46:0/1420683003 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c15d2a0 con 0x7f0f2c071d10 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f2bfff640 1 -- 172.21.15.46:0/1420683003 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979306715 0 0) 0x7f0f2c15db40 con 0x7f0f2c072d80 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.719+0000 7f0f31e3d640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c06e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.720+0000 7f0f30e3b640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c071d10 0x7f0f2c0d0690 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0f200097a0 tx=0x7f0f2002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900dc7ac312b1b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.720+0000 7f0f2bfff640 1 -- 172.21.15.46:0/1420683003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f200315a0 con 0x7f0f2c071d10 2024-04-15T15:16:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.720+0000 7f0f2bfff640 1 -- 172.21.15.46:0/1420683003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f20037070 con 0x7f0f2c071d10 2024-04-15T15:16:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.720+0000 7f0f2bfff640 1 -- 172.21.15.46:0/1420683003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20031960 con 0x7f0f2c071d10 2024-04-15T15:16:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 -- 172.21.15.46:0/1420683003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c071d10 msgr2=0x7f0f2c0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c071d10 0x7f0f2c0d0690 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0f200097a0 tx=0x7f0f2002b840 comp rx=0 tx=0).stop 2024-04-15T15:16:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 -- 172.21.15.46:0/1420683003 shutdown_connections 2024-04-15T15:16:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c071d10 0x7f0f2c0d0690 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 --2- 172.21.15.46:0/1420683003 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c06e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 -- 172.21.15.46:0/1420683003 >> 172.21.15.46:0/1420683003 conn(0x7f0f2c06be00 msgr2=0x7f0f2c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 -- 172.21.15.46:0/1420683003 shutdown_connections 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.721+0000 7f0f338a0640 1 -- 172.21.15.46:0/1420683003 wait complete. 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.722+0000 7f0f338a0640 1 Processor -- start 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.722+0000 7f0f338a0640 1 -- start start 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f338a0640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f2c071d10 0x7f0f2c159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f338a0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 0x7f0f2c14f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f338a0640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c14ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f338a0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c0afaf0 con 0x7f0f2c072d80 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f338a0640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2c15d860 con 0x7f0f2c071d10 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f338a0640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0f2c15cb90 con 0x7f0f2c0d0bd0 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 0x7f0f2c14f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 0x7f0f2c14f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41436/0 (socket says 172.21.15.46:41436) 2024-04-15T15:16:42.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 -- 172.21.15.46:0/4047453751 learned_addr learned my addr 172.21.15.46:0/4047453751 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f31e3d640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c14ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f3163c640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f2c071d10 0x7f0f2c159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 -- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 msgr2=0x7f0f2c14ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c14ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 -- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f2c071d10 msgr2=0x7f0f2c159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f2c071d10 0x7f0f2c159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f30e3b640 1 -- 172.21.15.46:0/4047453751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2c14f960 con 0x7f0f2c072d80 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.723+0000 7f0f31e3d640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c14ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.724+0000 7f0f30e3b640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 0x7f0f2c14f1f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0f20034040 tx=0x7f0f200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.724+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20004230 con 0x7f0f2c072d80 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.724+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f200043f0 con 0x7f0f2c072d80 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.724+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f20003850 con 0x7f0f2c072d80 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.724+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2c14fc40 con 0x7f0f2c072d80 2024-04-15T15:16:42.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.724+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2c150200 con 0x7f0f2c072d80 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.725+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2c0689e0 con 0x7f0f2c072d80 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.725+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f200039f0 con 0x7f0f2c072d80 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.726+0000 7f0f29ffb640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f1806c8f0 0x7f0f1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.726+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0f2002f030 con 0x7f0f2c072d80 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.726+0000 7f0f3163c640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f1806c8f0 0x7f0f1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.727+0000 7f0f3163c640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f1806c8f0 0x7f0f1806edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0012e0 tx=0x7f0f1c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.730+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f200b3020 con 0x7f0f2c072d80 2024-04-15T15:16:42.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.919+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0f2c0736d0 con 0x7f0f2c072d80 2024-04-15T15:16:42.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.919+0000 7f0f29ffb640 1 -- 172.21.15.46:0/4047453751 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f0f20041070 con 0x7f0f2c072d80 2024-04-15T15:16:42.921 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:42.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f1806c8f0 msgr2=0x7f0f1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f1806c8f0 0x7f0f1806edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0012e0 tx=0x7f0f1c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:42.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 msgr2=0x7f0f2c14f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 0x7f0f2c14f1f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0f20034040 tx=0x7f0f200046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 shutdown_connections 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f1806c8f0 0x7f0f1806edb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f2c0d0bd0 0x7f0f2c14ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c072d80 0x7f0f2c14f1f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 --2- 172.21.15.46:0/4047453751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f2c071d10 0x7f0f2c159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 >> 172.21.15.46:0/4047453751 conn(0x7f0f2c06be00 msgr2=0x7f0f2c15b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 shutdown_connections 2024-04-15T15:16:42.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:42.921+0000 7f0f338a0640 1 -- 172.21.15.46:0/4047453751 wait complete. 2024-04-15T15:16:42.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:43.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.040+0000 7f949097c640 1 Processor -- start 2024-04-15T15:16:43.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.040+0000 7f949097c640 1 -- start start 2024-04-15T15:16:43.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f949097c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f948c071d10 0x7f948c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f949097c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f948c0d0bd0 0x7f948c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f949097c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948c15db40 con 0x7f948c072d80 2024-04-15T15:16:43.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f949097c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f948c0d1040 con 0x7f948c0d0bd0 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f949097c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f948c072170 con 0x7f948c071d10 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f948c0d0bd0 0x7f948c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f948c072d80 0x7f948c0af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40398/0 (socket says 172.21.15.46:40398) 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f948c0d0bd0 0x7f948c06e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41444/0 (socket says 172.21.15.46:41444) 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948a59c640 1 -- 172.21.15.46:0/3341970440 learned_addr learned my addr 172.21.15.46:0/3341970440 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948ad9d640 1 -- 172.21.15.46:0/3341970440 >> v2:172.21.15.78:3300/0 conn(0x7f948c071d10 msgr2=0x7f948c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f9489d9b640 1 --2- 172.21.15.46:0/3341970440 >> v2:172.21.15.78:3300/0 conn(0x7f948c071d10 0x7f948c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948ad9d640 1 --2- 172.21.15.46:0/3341970440 >> v2:172.21.15.78:3300/0 conn(0x7f948c071d10 0x7f948c0d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948ad9d640 1 -- 172.21.15.46:0/3341970440 >> v1:172.21.15.46:6789/0 conn(0x7f948c072d80 legacy=0x7f948c0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948959a640 1 -- 172.21.15.46:0/3341970440 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2041511795 0 0) 0x7f948c15db40 con 0x7f948c072d80 2024-04-15T15:16:43.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.041+0000 7f948ad9d640 1 -- 172.21.15.46:0/3341970440 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948c15cfc0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.042+0000 7f948ad9d640 1 --2- 172.21.15.46:0/3341970440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c06e060 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f948c0d1240 tx=0x7f948402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f5becf31b0df29 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.042+0000 7f948959a640 1 -- 172.21.15.46:0/3341970440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94840315a0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.042+0000 7f948959a640 1 -- 172.21.15.46:0/3341970440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9484037070 con 0x7f948c0d0bd0 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.042+0000 7f948959a640 1 -- 172.21.15.46:0/3341970440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484031960 con 0x7f948c0d0bd0 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 -- 172.21.15.46:0/3341970440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 msgr2=0x7f948c06e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 --2- 172.21.15.46:0/3341970440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c06e060 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f948c0d1240 tx=0x7f948402b840 comp rx=0 tx=0).stop 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 -- 172.21.15.46:0/3341970440 shutdown_connections 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 --2- 172.21.15.46:0/3341970440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c06e060 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 --2- 172.21.15.46:0/3341970440 >> v2:172.21.15.78:3300/0 conn(0x7f948c071d10 0x7f948c0d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 -- 172.21.15.46:0/3341970440 >> 172.21.15.46:0/3341970440 conn(0x7f948c06be00 msgr2=0x7f948c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 -- 172.21.15.46:0/3341970440 shutdown_connections 2024-04-15T15:16:43.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.043+0000 7f949097c640 1 -- 172.21.15.46:0/3341970440 wait complete. 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.044+0000 7f949097c640 1 Processor -- start 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.044+0000 7f949097c640 1 -- start start 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f949097c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f948c071d10 0x7f948c150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f949097c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f948c072d80 0x7f948c0ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f949097c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c1594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f949097c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f948c15d860 con 0x7f948c0d0bd0 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f949097c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f948c15d580 con 0x7f948c071d10 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f949097c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f948c0afaf0 con 0x7f948c072d80 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f948ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c1594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f948ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c1594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41446/0 (socket says 172.21.15.46:41446) 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f948ad9d640 1 -- 172.21.15.46:0/2132004176 learned_addr learned my addr 172.21.15.46:0/2132004176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f9489d9b640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f948c072d80 0x7f948c0ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.045+0000 7f948a59c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f948c071d10 0x7f948c150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f948ad9d640 1 -- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f948c072d80 msgr2=0x7f948c0ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f948ad9d640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f948c072d80 0x7f948c0ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f948ad9d640 1 -- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f948c071d10 msgr2=0x7f948c150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f948ad9d640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f948c071d10 0x7f948c150680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f948ad9d640 1 -- 172.21.15.46:0/2132004176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948c0cd6d0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f9489d9b640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f948c072d80 0x7f948c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f948ad9d640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c1594c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f948402be50 tx=0x7f94840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484004230 con 0x7f948c0d0bd0 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94840043f0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948c0cd9b0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9484003850 con 0x7f948c0d0bd0 2024-04-15T15:16:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.046+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948c1571e0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.047+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f948c0689e0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.047+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f94840039f0 con 0x7f948c0d0bd0 2024-04-15T15:16:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.048+0000 7f9482ffd640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f946c06c8f0 0x7f946c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.048+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f948402f030 con 0x7f948c0d0bd0 2024-04-15T15:16:43.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.048+0000 7f948a59c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f946c06c8f0 0x7f946c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.049+0000 7f948a59c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f946c06c8f0 0x7f946c06edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f947c0012e0 tx=0x7f947c007ed0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.052+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9484041070 con 0x7f948c0d0bd0 2024-04-15T15:16:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.234+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f948c0b1720 con 0x7f948c0d0bd0 2024-04-15T15:16:43.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.235+0000 7f9482ffd640 1 -- 172.21.15.46:0/2132004176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f9484044340 con 0x7f948c0d0bd0 2024-04-15T15:16:43.236 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f946c06c8f0 msgr2=0x7f946c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f946c06c8f0 0x7f946c06edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f947c0012e0 tx=0x7f947c007ed0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 msgr2=0x7f948c1594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c1594c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f948402be50 tx=0x7f94840046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 shutdown_connections 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f946c06c8f0 0x7f946c06edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f948c0d0bd0 0x7f948c1594c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f948c072d80 0x7f948c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 --2- 172.21.15.46:0/2132004176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f948c071d10 0x7f948c150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 >> 172.21.15.46:0/2132004176 conn(0x7f948c06be00 msgr2=0x7f948c074df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 shutdown_connections 2024-04-15T15:16:43.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.237+0000 7f949097c640 1 -- 172.21.15.46:0/2132004176 wait complete. 2024-04-15T15:16:43.250 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:43.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 Processor -- start 2024-04-15T15:16:43.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 -- start start 2024-04-15T15:16:43.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f52300b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52300d2360 0x7f52300d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52300b47a0 con 0x7f523015eae0 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f52300b53b0 con 0x7f52300d2360 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.354+0000 7f5236493640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f52300b4a80 con 0x7f523015e700 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5227fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52300d2360 0x7f52300d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5234a30640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f523015eae0 0x7f52300cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40402/0 (socket says 172.21.15.46:40402) 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5227fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52300d2360 0x7f52300d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41448/0 (socket says 172.21.15.46:41448) 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5234a30640 1 -- 172.21.15.46:0/2916242649 learned_addr learned my addr 172.21.15.46:0/2916242649 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f522ffff640 1 --2- 172.21.15.46:0/2916242649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f52300b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5227fff640 1 -- 172.21.15.46:0/2916242649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 msgr2=0x7f52300b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5227fff640 1 --2- 172.21.15.46:0/2916242649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f52300b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5227fff640 1 -- 172.21.15.46:0/2916242649 >> v1:172.21.15.46:6789/0 conn(0x7f523015eae0 legacy=0x7f52300cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f5227fff640 1 -- 172.21.15.46:0/2916242649 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52300b4d60 con 0x7f52300d2360 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.355+0000 7f522f7fe640 1 -- 172.21.15.46:0/2916242649 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343282026 0 0) 0x7f52300b47a0 con 0x7f523015eae0 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5227fff640 1 --2- 172.21.15.46:0/2916242649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52300d2360 0x7f52300d2b60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f52180097a0 tx=0x7f521802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb6b1594790c49d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f522f7fe640 1 -- 172.21.15.46:0/2916242649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218035020 con 0x7f52300d2360 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f522f7fe640 1 -- 172.21.15.46:0/2916242649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5218031830 con 0x7f52300d2360 2024-04-15T15:16:43.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f522f7fe640 1 -- 172.21.15.46:0/2916242649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5218031c00 con 0x7f52300d2360 2024-04-15T15:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5236493640 1 -- 172.21.15.46:0/2916242649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52300d2360 msgr2=0x7f52300d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5236493640 1 --2- 172.21.15.46:0/2916242649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52300d2360 0x7f52300d2b60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f52180097a0 tx=0x7f521802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5236493640 1 -- 172.21.15.46:0/2916242649 shutdown_connections 2024-04-15T15:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5236493640 1 --2- 172.21.15.46:0/2916242649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52300d2360 0x7f52300d2b60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5236493640 1 --2- 172.21.15.46:0/2916242649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f52300b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.356+0000 7f5236493640 1 -- 172.21.15.46:0/2916242649 >> 172.21.15.46:0/2916242649 conn(0x7f523008bf60 msgr2=0x7f52300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.357+0000 7f5236493640 1 -- 172.21.15.46:0/2916242649 shutdown_connections 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.357+0000 7f5236493640 1 -- 172.21.15.46:0/2916242649 wait complete. 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 Processor -- start 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 -- start start 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52300d2360 0x7f52301f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f5230148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 0x7f52301487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52300b5d80 con 0x7f523015eae0 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f52300b4450 con 0x7f52300d2360 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5236493640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f52300d1d40 con 0x7f523015e700 2024-04-15T15:16:43.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5234a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 0x7f52301487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.358+0000 7f5234a30640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 0x7f52301487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41460/0 (socket says 172.21.15.46:41460) 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 -- 172.21.15.46:0/2592013788 learned_addr learned my addr 172.21.15.46:0/2592013788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5227fff640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f5230148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f522ffff640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52300d2360 0x7f52301f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 -- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 msgr2=0x7f5230148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f5230148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 -- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52300d2360 msgr2=0x7f52301f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52300d2360 0x7f52301f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 -- 172.21.15.46:0/2592013788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52300ce240 con 0x7f523015eae0 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5234a30640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 0x7f52301487c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f522800c700 tx=0x7f522800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228018070 con 0x7f523015eae0 2024-04-15T15:16:43.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522800ce10 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5230160bd0 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522800a860 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.359+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5230161190 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.360+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52300689e0 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.361+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f522801b070 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.361+0000 7f522d7fa640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f521006c920 0x7f521006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.361+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f522808b410 con 0x7f523015eae0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.361+0000 7f522ffff640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f521006c920 0x7f521006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.362+0000 7f522ffff640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f521006c920 0x7f521006ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5230157b40 tx=0x7f5220008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.365+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52280594d0 con 0x7f523015eae0 2024-04-15T15:16:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.566+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f52300ba500 con 0x7f523015eae0 2024-04-15T15:16:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.566+0000 7f522d7fa640 1 -- 172.21.15.46:0/2592013788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f5228058dd0 con 0x7f523015eae0 2024-04-15T15:16:43.568 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f521006c920 msgr2=0x7f521006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f521006c920 0x7f521006ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5230157b40 tx=0x7f5220008040 comp rx=0 tx=0).stop 2024-04-15T15:16:43.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 msgr2=0x7f52301487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 0x7f52301487c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f522800c700 tx=0x7f522800cbc0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 shutdown_connections 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f521006c920 0x7f521006ede0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523015eae0 0x7f52301487c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f523015e700 0x7f5230148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 --2- 172.21.15.46:0/2592013788 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52300d2360 0x7f52301f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 >> 172.21.15.46:0/2592013788 conn(0x7f523008bf60 msgr2=0x7f52300d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 shutdown_connections 2024-04-15T15:16:43.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.568+0000 7f5236493640 1 -- 172.21.15.46:0/2592013788 wait complete. 2024-04-15T15:16:43.581 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:16:43.581 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:43.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.703+0000 7f4556ca5640 1 Processor -- start 2024-04-15T15:16:43.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.703+0000 7f4556ca5640 1 -- start start 2024-04-15T15:16:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4556ca5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f455015e790 0x7f45500d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4556ca5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500cfd20 0x7f45500d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4556ca5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45500d2480 con 0x7f45500d0f00 2024-04-15T15:16:43.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4556ca5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f45500d2db0 con 0x7f455015e790 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4556ca5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f45500d2760 con 0x7f45500cfd20 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f455015e790 0x7f45500d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f455015e790 0x7f45500d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41466/0 (socket says 172.21.15.46:41466) 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 -- 172.21.15.46:0/3329905726 learned_addr learned my addr 172.21.15.46:0/3329905726 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4555242640 1 --1- 172.21.15.46:0/3329905726 >> v1:172.21.15.46:6789/0 conn(0x7f45500d0f00 0x7f45500b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40412/0 (socket says 172.21.15.46:40412) 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f454ffff640 1 --2- 172.21.15.46:0/3329905726 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500cfd20 0x7f45500d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 -- 172.21.15.46:0/3329905726 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500cfd20 msgr2=0x7f45500d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 --2- 172.21.15.46:0/3329905726 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500cfd20 0x7f45500d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 -- 172.21.15.46:0/3329905726 >> v1:172.21.15.46:6789/0 conn(0x7f45500d0f00 legacy=0x7f45500b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f4554a41640 1 -- 172.21.15.46:0/3329905726 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45500d2a40 con 0x7f455015e790 2024-04-15T15:16:43.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.704+0000 7f454f7fe640 1 -- 172.21.15.46:0/3329905726 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2065482313 0 0) 0x7f45500d2480 con 0x7f45500d0f00 2024-04-15T15:16:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.705+0000 7f4554a41640 1 --2- 172.21.15.46:0/3329905726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f45500d3240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4544009870 tx=0x7f454402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f28a8610fec5b5d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.705+0000 7f454f7fe640 1 -- 172.21.15.46:0/3329905726 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544034070 con 0x7f455015e790 2024-04-15T15:16:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.705+0000 7f454f7fe640 1 -- 172.21.15.46:0/3329905726 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4544030840 con 0x7f455015e790 2024-04-15T15:16:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.705+0000 7f454f7fe640 1 -- 172.21.15.46:0/3329905726 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544030c10 con 0x7f455015e790 2024-04-15T15:16:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 -- 172.21.15.46:0/3329905726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 msgr2=0x7f45500d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 --2- 172.21.15.46:0/3329905726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f45500d3240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4544009870 tx=0x7f454402b980 comp rx=0 tx=0).stop 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 -- 172.21.15.46:0/3329905726 shutdown_connections 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 --2- 172.21.15.46:0/3329905726 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500cfd20 0x7f45500d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 --2- 172.21.15.46:0/3329905726 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f45500d3240 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 -- 172.21.15.46:0/3329905726 >> 172.21.15.46:0/3329905726 conn(0x7f455008bf60 msgr2=0x7f45500d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 -- 172.21.15.46:0/3329905726 shutdown_connections 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.706+0000 7f4556ca5640 1 -- 172.21.15.46:0/3329905726 wait complete. 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.707+0000 7f4556ca5640 1 Processor -- start 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.707+0000 7f4556ca5640 1 -- start start 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4556ca5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45500cfd20 0x7f45500c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4556ca5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500d0f00 0x7f455015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4556ca5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f455014f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4556ca5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f455015ecf0 con 0x7f455015e790 2024-04-15T15:16:43.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4556ca5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f455015eb70 con 0x7f45500cfd20 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4556ca5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f45500d13b0 con 0x7f45500d0f00 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f455014f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f455014f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41474/0 (socket says 172.21.15.46:41474) 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 -- 172.21.15.46:0/4093215860 learned_addr learned my addr 172.21.15.46:0/4093215860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 -- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500d0f00 msgr2=0x7f455015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f454ffff640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500d0f00 0x7f455015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500d0f00 0x7f455015b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4554a41640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45500cfd20 0x7f45500c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 -- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45500cfd20 msgr2=0x7f45500c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45500cfd20 0x7f45500c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.708+0000 7f4555242640 1 -- 172.21.15.46:0/4093215860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45500c88c0 con 0x7f455015e790 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.709+0000 7f4555242640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f455014f6e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f454800c7a0 tx=0x7f454800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.709+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4548018070 con 0x7f455015e790 2024-04-15T15:16:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.709+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454800a5e0 con 0x7f455015e790 2024-04-15T15:16:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.709+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45500c8ba0 con 0x7f455015e790 2024-04-15T15:16:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.709+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454800aa50 con 0x7f455015e790 2024-04-15T15:16:43.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.709+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45500c9160 con 0x7f455015e790 2024-04-15T15:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.710+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45500689e0 con 0x7f455015e790 2024-04-15T15:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.710+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f454801b070 con 0x7f455015e790 2024-04-15T15:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.711+0000 7f454d7fa640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f453406c8f0 0x7f453406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.711+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f454808b550 con 0x7f455015e790 2024-04-15T15:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.711+0000 7f4554a41640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f453406c8f0 0x7f453406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:43.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.711+0000 7f4554a41640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f453406c8f0 0x7f453406edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4544036000 tx=0x7f454403b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:43.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.715+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4548059580 con 0x7f455015e790 2024-04-15T15:16:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.901+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45500d1ce0 con 0x7f455015e790 2024-04-15T15:16:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.902+0000 7f454d7fa640 1 -- 172.21.15.46:0/4093215860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f4548058e80 con 0x7f455015e790 2024-04-15T15:16:43.904 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:43.904 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.904+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f453406c8f0 msgr2=0x7f453406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.904+0000 7f4556ca5640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f453406c8f0 0x7f453406edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4544036000 tx=0x7f454403b040 comp rx=0 tx=0).stop 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.904+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 msgr2=0x7f455014f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.904+0000 7f4556ca5640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f455014f6e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f454800c7a0 tx=0x7f454800cc60 comp rx=0 tx=0).stop 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 shutdown_connections 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f453406c8f0 0x7f453406edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f455015e790 0x7f455014f6e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f45500d0f00 0x7f455015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 --2- 172.21.15.46:0/4093215860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45500cfd20 0x7f45500c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 >> 172.21.15.46:0/4093215860 conn(0x7f455008bf60 msgr2=0x7f45500caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 shutdown_connections 2024-04-15T15:16:43.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:43.905+0000 7f4556ca5640 1 -- 172.21.15.46:0/4093215860 wait complete. 2024-04-15T15:16:43.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:43.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.025+0000 7fcd4c500640 1 Processor -- start 2024-04-15T15:16:44.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.025+0000 7fcd4c500640 1 -- start start 2024-04-15T15:16:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4c500640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd440d15e0 0x7fcd440d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4c500640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd440d4a40 0x7fcd440d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4c500640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd4415eb70 con 0x7fcd4415e790 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4c500640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcd440d3790 con 0x7fcd440d4a40 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4c500640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcd440d45b0 con 0x7fcd440d15e0 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd440d4a40 0x7fcd440d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd440d4a40 0x7fcd440d4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41486/0 (socket says 172.21.15.46:41486) 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 -- 172.21.15.46:0/1962361628 learned_addr learned my addr 172.21.15.46:0/1962361628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd49a9b640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd440d15e0 0x7fcd440d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 -- 172.21.15.46:0/1962361628 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd440d15e0 msgr2=0x7fcd440d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd440d15e0 0x7fcd440d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 -- 172.21.15.46:0/1962361628 >> v1:172.21.15.78:6789/0 conn(0x7fcd4415e790 legacy=0x7fcd440b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.026+0000 7fcd4aa9d640 1 -- 172.21.15.46:0/1962361628 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd440d3c80 con 0x7fcd440d4a40 2024-04-15T15:16:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.027+0000 7fcd49a9b640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd440d15e0 0x7fcd440d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.027+0000 7fcd4aa9d640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d4a40 0x7fcd440d4eb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcd440d3910 tx=0x7fcd3c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60163deb281ed1bc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.027+0000 7fcd4929a640 1 -- 172.21.15.46:0/1962361628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c038070 con 0x7fcd440d4a40 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.027+0000 7fcd4929a640 1 -- 172.21.15.46:0/1962361628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3c031700 con 0x7fcd440d4a40 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.027+0000 7fcd4929a640 1 -- 172.21.15.46:0/1962361628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c031ad0 con 0x7fcd440d4a40 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 -- 172.21.15.46:0/1962361628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d4a40 msgr2=0x7fcd440d4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d4a40 0x7fcd440d4eb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcd440d3910 tx=0x7fcd3c02b7a0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 -- 172.21.15.46:0/1962361628 shutdown_connections 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d4a40 0x7fcd440d4eb0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1962361628 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd440d15e0 0x7fcd440d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 -- 172.21.15.46:0/1962361628 >> 172.21.15.46:0/1962361628 conn(0x7fcd44067b40 msgr2=0x7fcd44067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 -- 172.21.15.46:0/1962361628 shutdown_connections 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.028+0000 7fcd4c500640 1 -- 172.21.15.46:0/1962361628 wait complete. 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.029+0000 7fcd4c500640 1 Processor -- start 2024-04-15T15:16:44.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.029+0000 7fcd4c500640 1 -- start start 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4c500640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 0x7fcd440c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4c500640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd440d4a40 0x7fcd441611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4c500640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd4415e790 0x7fcd4414c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4c500640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd440d4240 con 0x7fcd440d15e0 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4c500640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcd440d53f0 con 0x7fcd440d4a40 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4c500640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcd440b7db0 con 0x7fcd4415e790 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 0x7fcd440c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 0x7fcd440c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41496/0 (socket says 172.21.15.46:41496) 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 -- 172.21.15.46:0/1882165037 learned_addr learned my addr 172.21.15.46:0/1882165037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4aa9d640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd4415e790 0x7fcd4414c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd49a9b640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd440d4a40 0x7fcd441611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 -- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd4415e790 msgr2=0x7fcd4414c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd4415e790 0x7fcd4414c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 -- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd440d4a40 msgr2=0x7fcd441611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd440d4a40 0x7fcd441611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4a29c640 1 -- 172.21.15.46:0/1882165037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd440c89b0 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.030+0000 7fcd4aa9d640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd4415e790 0x7fcd4414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.031+0000 7fcd4a29c640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 0x7fcd440c8120 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400c7c0 tx=0x7fcd3400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.031+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34016020 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.031+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3400aa50 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.031+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd440c8c90 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.031+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34012420 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.031+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd440c8660 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.032+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd440689e0 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.032+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd34019020 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.033+0000 7fcd42ffd640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd3006c920 0x7fcd3006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.033+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcd3408bd30 con 0x7fcd440d15e0 2024-04-15T15:16:44.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.033+0000 7fcd49a9b640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd3006c920 0x7fcd3006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.033+0000 7fcd49a9b640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd3006c920 0x7fcd3006ede0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcd4415d8e0 tx=0x7fcd38008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.037+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd34020070 con 0x7fcd440d15e0 2024-04-15T15:16:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.220+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd440d1c50 con 0x7fcd440d15e0 2024-04-15T15:16:44.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.221+0000 7fcd42ffd640 1 -- 172.21.15.46:0/1882165037 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fcd34059d60 con 0x7fcd440d15e0 2024-04-15T15:16:44.223 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd3006c920 msgr2=0x7fcd3006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd3006c920 0x7fcd3006ede0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcd4415d8e0 tx=0x7fcd38008040 comp rx=0 tx=0).stop 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 msgr2=0x7fcd440c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 0x7fcd440c8120 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcd3400c7c0 tx=0x7fcd3400cc80 comp rx=0 tx=0).stop 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 shutdown_connections 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd3006c920 0x7fcd3006ede0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd4415e790 0x7fcd4414c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd440d4a40 0x7fcd441611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 --2- 172.21.15.46:0/1882165037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd440d15e0 0x7fcd440c8120 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 >> 172.21.15.46:0/1882165037 conn(0x7fcd44067b40 msgr2=0x7fcd440b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 shutdown_connections 2024-04-15T15:16:44.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.223+0000 7fcd4c500640 1 -- 172.21.15.46:0/1882165037 wait complete. 2024-04-15T15:16:44.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.341+0000 7f6337822640 1 Processor -- start 2024-04-15T15:16:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.342+0000 7f6337822640 1 -- start start 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.342+0000 7f6337822640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63300cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.342+0000 7f6337822640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63300b47a0 con 0x7f63300d2360 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.342+0000 7f6337822640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63300b4a80 con 0x7f633015e700 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.342+0000 7f6337822640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f63300b53b0 con 0x7f63300d3270 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.342+0000 7f6334dbd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f63300d2360 0x7f63300d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40420/0 (socket says 172.21.15.46:40420) 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f6334dbd640 1 -- 172.21.15.46:0/1533337490 learned_addr learned my addr 172.21.15.46:0/1533337490 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f6335dbf640 1 --2- 172.21.15.46:0/1533337490 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63300cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223766622 0 0) 0x7f63300b47a0 con 0x7f63300d2360 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6318003520 con 0x7f63300d2360 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2291711547 0 0) 0x7f63300b4a80 con 0x7f633015e700 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63300b47a0 con 0x7f633015e700 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f6335dbf640 1 -- 172.21.15.46:0/1533337490 >> v1:172.21.15.78:6789/0 conn(0x7f633015e700 legacy=0x7f63300b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1028432829 0 0) 0x7f6318003520 con 0x7f63300d2360 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f6335dbf640 1 -- 172.21.15.46:0/1533337490 >> v1:172.21.15.46:6789/0 conn(0x7f63300d2360 legacy=0x7f63300d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.343+0000 7f6335dbf640 1 -- 172.21.15.46:0/1533337490 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63300b4d60 con 0x7f63300d3270 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.344+0000 7f6335dbf640 1 --2- 172.21.15.46:0/1533337490 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63300cf370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6320009dc0 tx=0x7f632002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9596716739baf07 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.344+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320039070 con 0x7f63300d3270 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.344+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f632002c910 con 0x7f63300d3270 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.344+0000 7f632ffff640 1 -- 172.21.15.46:0/1533337490 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324001670 con 0x7f63300d3270 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 -- 172.21.15.46:0/1533337490 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 msgr2=0x7f63300cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 --2- 172.21.15.46:0/1533337490 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63300cf370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6320009dc0 tx=0x7f632002bde0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 -- 172.21.15.46:0/1533337490 shutdown_connections 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 --2- 172.21.15.46:0/1533337490 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63300cf370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 -- 172.21.15.46:0/1533337490 >> 172.21.15.46:0/1533337490 conn(0x7f633008bf60 msgr2=0x7f63300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 -- 172.21.15.46:0/1533337490 shutdown_connections 2024-04-15T15:16:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.345+0000 7f6337822640 1 -- 172.21.15.46:0/1533337490 wait complete. 2024-04-15T15:16:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.346+0000 7f6337822640 1 Processor -- start 2024-04-15T15:16:44.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.346+0000 7f6337822640 1 -- start start 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63300d2360 0x7f63300b9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63301f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 0x7f63301f8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63300d1d40 con 0x7f633015e700 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f63300b4450 con 0x7f63300d2360 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f63300d1ec0 con 0x7f63300d3270 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 0x7f63301f8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 0x7f63301f8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41510/0 (socket says 172.21.15.46:41510) 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 -- 172.21.15.46:0/3666549240 learned_addr learned my addr 172.21.15.46:0/3666549240 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:44.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6334dbd640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63301f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f63355be640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63300d2360 0x7f63300b9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 -- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 msgr2=0x7f63301f7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63301f7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 -- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63300d2360 msgr2=0x7f63300b9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63300d2360 0x7f63300b9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 -- 172.21.15.46:0/3666549240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6330161830 con 0x7f633015e700 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6335dbf640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 0x7f63301f8510 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6320037040 tx=0x7f6320004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320039050 con 0x7f633015e700 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6320045070 con 0x7f633015e700 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6330161b10 con 0x7f633015e700 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.347+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320009450 con 0x7f633015e700 2024-04-15T15:16:44.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.348+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f633014e9e0 con 0x7f633015e700 2024-04-15T15:16:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.349+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63300689e0 con 0x7f633015e700 2024-04-15T15:16:44.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.349+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6320040690 con 0x7f633015e700 2024-04-15T15:16:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.350+0000 7f632dffb640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f631c06c920 0x7f631c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.350+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f63200ae140 con 0x7f633015e700 2024-04-15T15:16:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.350+0000 7f63355be640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f631c06c920 0x7f631c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.350+0000 7f63355be640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f631c06c920 0x7f631c06ede0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6328001230 tx=0x7f6328009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.354+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f632004e020 con 0x7f633015e700 2024-04-15T15:16:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.538+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f63300ce680 con 0x7f633015e700 2024-04-15T15:16:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.539+0000 7f632dffb640 1 -- 172.21.15.46:0/3666549240 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f632007bb40 con 0x7f633015e700 2024-04-15T15:16:44.540 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f631c06c920 msgr2=0x7f631c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f631c06c920 0x7f631c06ede0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6328001230 tx=0x7f6328009040 comp rx=0 tx=0).stop 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 msgr2=0x7f63301f8510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 0x7f63301f8510 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6320037040 tx=0x7f6320004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 shutdown_connections 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f631c06c920 0x7f631c06ede0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f633015e700 0x7f63301f8510 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f63300d3270 0x7f63301f7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 --2- 172.21.15.46:0/3666549240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63300d2360 0x7f63300b9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 >> 172.21.15.46:0/3666549240 conn(0x7f633008bf60 msgr2=0x7f63300d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 shutdown_connections 2024-04-15T15:16:44.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.541+0000 7f6337822640 1 -- 172.21.15.46:0/3666549240 wait complete. 2024-04-15T15:16:44.554 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.659+0000 7fae97ba8640 1 Processor -- start 2024-04-15T15:16:44.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.659+0000 7fae97ba8640 1 -- start start 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae97ba8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90070940 0x7fae9009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae97ba8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae90160310 0x7fae90070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae97ba8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae900d05e0 con 0x7fae900d0da0 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae97ba8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fae90071260 con 0x7fae90160310 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae97ba8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae900d1940 con 0x7fae90070940 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae90160310 0x7fae90070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae90160310 0x7fae90070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41524/0 (socket says 172.21.15.46:41524) 2024-04-15T15:16:44.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 -- 172.21.15.46:0/3621300415 learned_addr learned my addr 172.21.15.46:0/3621300415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 -- 172.21.15.46:0/3621300415 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90070940 msgr2=0x7fae9009b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95944640 1 --2- 172.21.15.46:0/3621300415 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90070940 0x7fae9009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 --2- 172.21.15.46:0/3621300415 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90070940 0x7fae9009b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 -- 172.21.15.46:0/3621300415 >> v1:172.21.15.78:6789/0 conn(0x7fae900d0da0 legacy=0x7fae900d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.660+0000 7fae95143640 1 -- 172.21.15.46:0/3621300415 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9015f720 con 0x7fae90160310 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.661+0000 7fae95143640 1 --2- 172.21.15.46:0/3621300415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae90160310 0x7fae90070d20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fae80009870 tx=0x7fae8002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cae4be7fb5ef59 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.661+0000 7fae94942640 1 -- 172.21.15.46:0/3621300415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80035020 con 0x7fae90160310 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.661+0000 7fae94942640 1 -- 172.21.15.46:0/3621300415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae80031640 con 0x7fae90160310 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.661+0000 7fae94942640 1 -- 172.21.15.46:0/3621300415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8002d040 con 0x7fae90160310 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 -- 172.21.15.46:0/3621300415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae90160310 msgr2=0x7fae90070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 --2- 172.21.15.46:0/3621300415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae90160310 0x7fae90070d20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fae80009870 tx=0x7fae8002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 -- 172.21.15.46:0/3621300415 shutdown_connections 2024-04-15T15:16:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 --2- 172.21.15.46:0/3621300415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae90160310 0x7fae90070d20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 --2- 172.21.15.46:0/3621300415 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90070940 0x7fae9009b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 -- 172.21.15.46:0/3621300415 >> 172.21.15.46:0/3621300415 conn(0x7fae900b3f40 msgr2=0x7fae900b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 -- 172.21.15.46:0/3621300415 shutdown_connections 2024-04-15T15:16:44.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.662+0000 7fae97ba8640 1 -- 172.21.15.46:0/3621300415 wait complete. 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.663+0000 7fae97ba8640 1 Processor -- start 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.663+0000 7fae97ba8640 1 -- start start 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae97ba8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90070940 0x7fae900c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae97ba8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 0x7fae900c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae97ba8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90160310 0x7fae900c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae97ba8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae900d0880 con 0x7fae900d0da0 2024-04-15T15:16:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae97ba8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fae9015fce0 con 0x7fae90070940 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae97ba8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae90160770 con 0x7fae90160310 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 0x7fae900c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 0x7fae900c89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41528/0 (socket says 172.21.15.46:41528) 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 -- 172.21.15.46:0/1172279283 learned_addr learned my addr 172.21.15.46:0/1172279283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae96145640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90160310 0x7fae900c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95944640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90070940 0x7fae900c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 -- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90160310 msgr2=0x7fae900c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90160310 0x7fae900c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 -- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90070940 msgr2=0x7fae900c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90070940 0x7fae900c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 -- 172.21.15.46:0/1172279283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae90156700 con 0x7fae900d0da0 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.664+0000 7fae95143640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 0x7fae900c89a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.665+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80040070 con 0x7fae900d0da0 2024-04-15T15:16:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.665+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae80039bf0 con 0x7fae900d0da0 2024-04-15T15:16:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.665+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae901569e0 con 0x7fae900d0da0 2024-04-15T15:16:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.665+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80035040 con 0x7fae900d0da0 2024-04-15T15:16:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.665+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae90156fa0 con 0x7fae900d0da0 2024-04-15T15:16:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.666+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae901572e0 con 0x7fae900d0da0 2024-04-15T15:16:44.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.666+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae8004b060 con 0x7fae900d0da0 2024-04-15T15:16:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.667+0000 7fae867fc640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae7406c8f0 0x7fae7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.667+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fae800ad9b0 con 0x7fae900d0da0 2024-04-15T15:16:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.667+0000 7fae95944640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae7406c8f0 0x7fae7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.667+0000 7fae95944640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae7406c8f0 0x7fae7406edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae8c006fd0 tx=0x7fae8c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.671+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae80031610 con 0x7fae900d0da0 2024-04-15T15:16:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.855+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fae900689e0 con 0x7fae900d0da0 2024-04-15T15:16:44.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.855+0000 7fae867fc640 1 -- 172.21.15.46:0/1172279283 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fae8007ba70 con 0x7fae900d0da0 2024-04-15T15:16:44.857 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae7406c8f0 msgr2=0x7fae7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae7406c8f0 0x7fae7406edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fae8c006fd0 tx=0x7fae8c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 msgr2=0x7fae900c89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 0x7fae900c89a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fae80000c00 tx=0x7fae800046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 shutdown_connections 2024-04-15T15:16:44.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae7406c8f0 0x7fae7406edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae90160310 0x7fae900c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae900d0da0 0x7fae900c89a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 --2- 172.21.15.46:0/1172279283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90070940 0x7fae900c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 >> 172.21.15.46:0/1172279283 conn(0x7fae900b3f40 msgr2=0x7fae900b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 shutdown_connections 2024-04-15T15:16:44.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.857+0000 7fae97ba8640 1 -- 172.21.15.46:0/1172279283 wait complete. 2024-04-15T15:16:44.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.975+0000 7fa764455640 1 Processor -- start 2024-04-15T15:16:44.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.976+0000 7fa764455640 1 -- start start 2024-04-15T15:16:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa764455640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa75c15ec70 0x7fa75c0afd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa764455640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 0x7fa75c0cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa764455640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa75c0aecf0 con 0x7fa75c0ae790 2024-04-15T15:16:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa764455640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa75c0aefd0 con 0x7fa75c15ec70 2024-04-15T15:16:44.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa764455640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa75c0af2b0 con 0x7fa75c0ae330 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7629f2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa75c0ae790 0x7fa75c0b4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40426/0 (socket says 172.21.15.46:40426) 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7629f2640 1 -- 172.21.15.46:0/658272262 learned_addr learned my addr 172.21.15.46:0/658272262 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7619f0640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 0x7fa75c0cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7621f1640 1 --2- 172.21.15.46:0/658272262 >> v2:172.21.15.78:3300/0 conn(0x7fa75c15ec70 0x7fa75c0afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051032552 0 0) 0x7fa75c0aecf0 con 0x7fa75c0ae790 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa744003520 con 0x7fa75c0ae790 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1689268957 0 0) 0x7fa744003520 con 0x7fa75c0ae790 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 msgr2=0x7fa75c0cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 0x7fa75c0cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 >> v2:172.21.15.78:3300/0 conn(0x7fa75c15ec70 msgr2=0x7fa75c0afd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 --2- 172.21.15.46:0/658272262 >> v2:172.21.15.78:3300/0 conn(0x7fa75c15ec70 0x7fa75c0afd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c0af590 con 0x7fa75c0ae790 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fa754001670 con 0x7fa75c0ae790 2024-04-15T15:16:44.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 >> v1:172.21.15.46:6789/0 conn(0x7fa75c0ae790 legacy=0x7fa75c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 0x7fa7440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa744003f30 0x7fa744004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744004d40 0x7fa7440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c0aefd0 con 0x7fa744003f30 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c0af2b0 con 0x7fa744004d40 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.977+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa74400ba10 con 0x7fa75c0ae330 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7619f0640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa744003f30 0x7fa744004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7621f1640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 0x7fa7440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7629f2640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744004d40 0x7fa7440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7619f0640 1 -- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 msgr2=0x7fa7440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7619f0640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa75c0ae330 0x7fa7440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7619f0640 1 -- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744004d40 msgr2=0x7fa7440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7619f0640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa744004d40 0x7fa7440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.978+0000 7fa7619f0640 1 -- 172.21.15.46:0/658272262 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa744003520 con 0x7fa744003f30 2024-04-15T15:16:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa7619f0640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa744003f30 0x7fa744004800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa750009af0 tx=0x7fa744005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0e76316caa3053 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750039070 con 0x7fa744003f30 2024-04-15T15:16:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75002c6b0 con 0x7fa744003f30 2024-04-15T15:16:44.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa7611ef640 1 -- 172.21.15.46:0/658272262 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750009690 con 0x7fa744003f30 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa764455640 1 -- 172.21.15.46:0/658272262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa744003f30 msgr2=0x7fa744004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa764455640 1 --2- 172.21.15.46:0/658272262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa744003f30 0x7fa744004800 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa750009af0 tx=0x7fa744005830 comp rx=0 tx=0).stop 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa7629f2640 1 -- 172.21.15.46:0/658272262 reap_dead start 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa764455640 1 -- 172.21.15.46:0/658272262 shutdown_connections 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa764455640 1 -- 172.21.15.46:0/658272262 >> 172.21.15.46:0/658272262 conn(0x7fa75c0b6870 msgr2=0x7fa75c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa764455640 1 -- 172.21.15.46:0/658272262 shutdown_connections 2024-04-15T15:16:44.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.979+0000 7fa764455640 1 -- 172.21.15.46:0/658272262 wait complete. 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 Processor -- start 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 -- start start 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa744003f30 0x7fa75c1522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 0x7fa75c157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa75c152930 0x7fa75c1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c0d1ce0 con 0x7fa75c1556d0 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c0d1b60 con 0x7fa75c152930 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa764455640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa75c0b4a60 con 0x7fa744003f30 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa7619f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 0x7fa75c157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa7619f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 0x7fa75c157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41542/0 (socket says 172.21.15.46:41542) 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.981+0000 7fa7619f0640 1 -- 172.21.15.46:0/138534410 learned_addr learned my addr 172.21.15.46:0/138534410 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7629f2640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa75c152930 0x7fa75c1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7621f1640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa744003f30 0x7fa75c1522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7619f0640 1 -- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa744003f30 msgr2=0x7fa75c1522a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7619f0640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa744003f30 0x7fa75c1522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7619f0640 1 -- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa75c152930 msgr2=0x7fa75c1575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7619f0640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa75c152930 0x7fa75c1575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7619f0640 1 -- 172.21.15.46:0/138534410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c156ce0 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7629f2640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa75c152930 0x7fa75c1575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa7619f0640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 0x7fa75c157060 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa750037040 tx=0x7fa750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750042020 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75003a040 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75c155c40 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750039070 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.982+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75c156200 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.983+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa75c1554d0 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.984+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa750045020 con 0x7fa75c1556d0 2024-04-15T15:16:44.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.984+0000 7fa75affd640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa74806c8f0 0x7fa74806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.984+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa7500acd20 con 0x7fa75c1556d0 2024-04-15T15:16:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.984+0000 7fa7621f1640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa74806c8f0 0x7fa74806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.985+0000 7fa7621f1640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa74806c8f0 0x7fa74806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa74c007440 tx=0x7fa74c00b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:44.988+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa75c1554d0 con 0x7fa75c1556d0 2024-04-15T15:16:45.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.173+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa75c0689e0 con 0x7fa75c1556d0 2024-04-15T15:16:45.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.174+0000 7fa75affd640 1 -- 172.21.15.46:0/138534410 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fa75007ade0 con 0x7fa75c1556d0 2024-04-15T15:16:45.176 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa74806c8f0 msgr2=0x7fa74806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa74806c8f0 0x7fa74806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa74c007440 tx=0x7fa74c00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 msgr2=0x7fa75c157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 0x7fa75c157060 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa750037040 tx=0x7fa750004730 comp rx=0 tx=0).stop 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 shutdown_connections 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa74806c8f0 0x7fa74806edb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa75c152930 0x7fa75c1575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa75c1556d0 0x7fa75c157060 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 --2- 172.21.15.46:0/138534410 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa744003f30 0x7fa75c1522a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 >> 172.21.15.46:0/138534410 conn(0x7fa75c0b6870 msgr2=0x7fa75c154dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:45.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 shutdown_connections 2024-04-15T15:16:45.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.176+0000 7fa764455640 1 -- 172.21.15.46:0/138534410 wait complete. 2024-04-15T15:16:45.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:45.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.293+0000 7f7c15e55640 1 Processor -- start 2024-04-15T15:16:45.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.293+0000 7f7c15e55640 1 -- start start 2024-04-15T15:16:45.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c15e55640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c101602d0 0x7f7c100cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c15e55640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c1009ab40 0x7f7c1009afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c15e55640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c1015dbd0 con 0x7f7c101606b0 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c15e55640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f7c100cd6a0 con 0x7f7c101602d0 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c15e55640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7c100cd980 con 0x7f7c1009ab40 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c0f7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c101602d0 0x7f7c100cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c0effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7c101606b0 0x7f7c100d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:51320/0 (socket says 172.21.15.46:51320) 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0effd640 1 -- 172.21.15.46:0/3816904903 learned_addr learned my addr 172.21.15.46:0/3816904903 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.294+0000 7f7c0ffff640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c1009ab40 0x7f7c1009afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c1009ab40 msgr2=0x7f7c1009afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0f7fe640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c1009ab40 0x7f7c1009afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3816904903 >> v1:172.21.15.78:6789/0 conn(0x7f7c101606b0 legacy=0x7f7c100d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0e7fc640 1 -- 172.21.15.46:0/3816904903 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265171454 0 0) 0x7f7c1015dbd0 con 0x7f7c101606b0 2024-04-15T15:16:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3816904903 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c100cdc60 con 0x7f7c101602d0 2024-04-15T15:16:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.295+0000 7f7c0ffff640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c1009ab40 0x7f7c1009afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.296+0000 7f7c0f7fe640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100cd000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c000097a0 tx=0x7f7c0002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=825755695df5bc97 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:45.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.296+0000 7f7c0e7fc640 1 -- 172.21.15.46:0/3816904903 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00035020 con 0x7f7c101602d0 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.296+0000 7f7c0e7fc640 1 -- 172.21.15.46:0/3816904903 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c00031830 con 0x7f7c101602d0 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.296+0000 7f7c0e7fc640 1 -- 172.21.15.46:0/3816904903 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00031c00 con 0x7f7c101602d0 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.296+0000 7f7c15e55640 1 -- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 msgr2=0x7f7c100cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.296+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100cd000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c000097a0 tx=0x7f7c0002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.297+0000 7f7c15e55640 1 -- 172.21.15.46:0/3816904903 shutdown_connections 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.297+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c1009ab40 0x7f7c1009afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.297+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3816904903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100cd000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.297+0000 7f7c15e55640 1 -- 172.21.15.46:0/3816904903 >> 172.21.15.46:0/3816904903 conn(0x7f7c100d2460 msgr2=0x7f7c100d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.297+0000 7f7c15e55640 1 -- 172.21.15.46:0/3816904903 shutdown_connections 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.297+0000 7f7c15e55640 1 -- 172.21.15.46:0/3816904903 wait complete. 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.298+0000 7f7c15e55640 1 Processor -- start 2024-04-15T15:16:45.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.298+0000 7f7c15e55640 1 -- start start 2024-04-15T15:16:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c15e55640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 0x7f7c101f3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c15e55640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c15e55640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c101606b0 0x7f7c100c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c15e55640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c10160b10 con 0x7f7c1009ab40 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c15e55640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c100ce220 con 0x7f7c101602d0 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c15e55640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1009b510 con 0x7f7c101606b0 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 0x7f7c101f3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 0x7f7c101f3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41552/0 (socket says 172.21.15.46:41552) 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3666739220 learned_addr learned my addr 172.21.15.46:0/3666739220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0ffff640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c101606b0 0x7f7c100c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0effd640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c101606b0 msgr2=0x7f7c100c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c101606b0 0x7f7c100c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 msgr2=0x7f7c100c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 -- 172.21.15.46:0/3666739220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c100c6ba0 con 0x7f7c1009ab40 2024-04-15T15:16:45.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.299+0000 7f7c0f7fe640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 0x7f7c101f3580 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7c0002bee0 tx=0x7f7c000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.300+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00035040 con 0x7f7c1009ab40 2024-04-15T15:16:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.300+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0003b860 con 0x7f7c1009ab40 2024-04-15T15:16:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.300+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c100c6e80 con 0x7f7c1009ab40 2024-04-15T15:16:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.300+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0003bc50 con 0x7f7c1009ab40 2024-04-15T15:16:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.300+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c100c7440 con 0x7f7c1009ab40 2024-04-15T15:16:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.301+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c100689e0 con 0x7f7c1009ab40 2024-04-15T15:16:45.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.301+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7c00038020 con 0x7f7c1009ab40 2024-04-15T15:16:45.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.301+0000 7f7bf7fff640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7bfc06c8f0 0x7f7bfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.302+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7c000ae050 con 0x7f7c1009ab40 2024-04-15T15:16:45.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.302+0000 7f7c0effd640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7bfc06c8f0 0x7f7bfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.302+0000 7f7c0effd640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7bfc06c8f0 0x7f7bfc06edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7c100b3ce0 tx=0x7f7c04007de0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.306+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c00031800 con 0x7f7c1009ab40 2024-04-15T15:16:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.503+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7c100cec50 con 0x7f7c1009ab40 2024-04-15T15:16:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.503+0000 7f7bf7fff640 1 -- 172.21.15.46:0/3666739220 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f7c0007bff0 con 0x7f7c1009ab40 2024-04-15T15:16:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7bfc06c8f0 msgr2=0x7f7bfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7bfc06c8f0 0x7f7bfc06edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7c100b3ce0 tx=0x7f7c04007de0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 msgr2=0x7f7c101f3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 0x7f7c101f3580 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7c0002bee0 tx=0x7f7c000046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 shutdown_connections 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7bfc06c8f0 0x7f7bfc06edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c101606b0 0x7f7c100c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c101602d0 0x7f7c100c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 --2- 172.21.15.46:0/3666739220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1009ab40 0x7f7c101f3580 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 >> 172.21.15.46:0/3666739220 conn(0x7f7c100d2460 msgr2=0x7f7c1015cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 shutdown_connections 2024-04-15T15:16:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.505+0000 7f7c15e55640 1 -- 172.21.15.46:0/3666739220 wait complete. 2024-04-15T15:16:45.518 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-04-15T15:16:45.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:45.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:45.553 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:45.553 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:16:45.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:45.554 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:16:45.612 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-15T15:16:45.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.715+0000 7f75fc06a640 1 Processor -- start 2024-04-15T15:16:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fc06a640 1 -- start start 2024-04-15T15:16:45.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fc06a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f75f415e700 0x7f75f40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fc06a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75f40d3270 0x7f75f40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fc06a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f40b47a0 con 0x7f75f40d2360 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fc06a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40b53b0 con 0x7f75f40d3270 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fc06a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f75f40b4a80 con 0x7f75f415e700 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75fa607640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f75f40d3270 0x7f75f40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75f9605640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f75f40d2360 0x7f75f40d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40432/0 (socket says 172.21.15.46:40432) 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.716+0000 7f75f9605640 1 -- 172.21.15.46:0/673610839 learned_addr learned my addr 172.21.15.46:0/673610839 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.717+0000 7f75f9e06640 1 --2- 172.21.15.46:0/673610839 >> v2:172.21.15.78:3300/0 conn(0x7f75f415e700 0x7f75f40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.717+0000 7f75fa607640 1 -- 172.21.15.46:0/673610839 >> v2:172.21.15.78:3300/0 conn(0x7f75f415e700 msgr2=0x7f75f40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.717+0000 7f75fa607640 1 --2- 172.21.15.46:0/673610839 >> v2:172.21.15.78:3300/0 conn(0x7f75f415e700 0x7f75f40b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.717+0000 7f75fa607640 1 -- 172.21.15.46:0/673610839 >> v1:172.21.15.46:6789/0 conn(0x7f75f40d2360 legacy=0x7f75f40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.717+0000 7f75fa607640 1 -- 172.21.15.46:0/673610839 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f40b4d60 con 0x7f75f40d3270 2024-04-15T15:16:45.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.717+0000 7f75f8e04640 1 -- 172.21.15.46:0/673610839 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2527101121 0 0) 0x7f75f40b47a0 con 0x7f75f40d2360 2024-04-15T15:16:45.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fa607640 1 --2- 172.21.15.46:0/673610839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f40cf370 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f75f406c6c0 tx=0x7f75e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d094fa008701ec16 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75f8e04640 1 -- 172.21.15.46:0/673610839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4039070 con 0x7f75f40d3270 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75f8e04640 1 -- 172.21.15.46:0/673610839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e402cc80 con 0x7f75f40d3270 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75f8e04640 1 -- 172.21.15.46:0/673610839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4034660 con 0x7f75f40d3270 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 -- 172.21.15.46:0/673610839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 msgr2=0x7f75f40cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 --2- 172.21.15.46:0/673610839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f40cf370 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f75f406c6c0 tx=0x7f75e402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 -- 172.21.15.46:0/673610839 shutdown_connections 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 --2- 172.21.15.46:0/673610839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f40cf370 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 --2- 172.21.15.46:0/673610839 >> v2:172.21.15.78:3300/0 conn(0x7f75f415e700 0x7f75f40b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 -- 172.21.15.46:0/673610839 >> 172.21.15.46:0/673610839 conn(0x7f75f408bf60 msgr2=0x7f75f40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 -- 172.21.15.46:0/673610839 shutdown_connections 2024-04-15T15:16:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.718+0000 7f75fc06a640 1 -- 172.21.15.46:0/673610839 wait complete. 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 Processor -- start 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 -- start start 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75f40d2360 0x7f75f40b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f4155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75f415e700 0x7f75f40c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40b5d80 con 0x7f75f40d3270 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f75f40b4450 con 0x7f75f40d2360 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75fc06a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f75f40d1d40 con 0x7f75f415e700 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75f9605640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75f9605640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f4155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41568/0 (socket says 172.21.15.46:41568) 2024-04-15T15:16:45.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.720+0000 7f75f9605640 1 -- 172.21.15.46:0/1128397465 learned_addr learned my addr 172.21.15.46:0/1128397465 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9e06640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75f40d2360 0x7f75f40b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75fa607640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75f415e700 0x7f75f40c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9605640 1 -- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75f415e700 msgr2=0x7f75f40c9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9605640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75f415e700 0x7f75f40c9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9605640 1 -- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75f40d2360 msgr2=0x7f75f40b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9605640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75f40d2360 0x7f75f40b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9605640 1 -- 172.21.15.46:0/1128397465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f415eb80 con 0x7f75f40d3270 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75fa607640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75f415e700 0x7f75f40c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f9605640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f4155220 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c4d0 tx=0x7f75e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8018070 con 0x7f75f40d3270 2024-04-15T15:16:45.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e8007500 con 0x7f75f40d3270 2024-04-15T15:16:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f415ee20 con 0x7f75f40d3270 2024-04-15T15:16:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8004a20 con 0x7f75f40d3270 2024-04-15T15:16:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.721+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f415f3e0 con 0x7f75f40d3270 2024-04-15T15:16:45.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.722+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f40689e0 con 0x7f75f40d3270 2024-04-15T15:16:45.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.723+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75e801b070 con 0x7f75f40d3270 2024-04-15T15:16:45.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.723+0000 7f75f27fc640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f75e006c8f0 0x7f75e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:45.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.723+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f75e808b1f0 con 0x7f75f40d3270 2024-04-15T15:16:45.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.723+0000 7f75f9e06640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f75e006c8f0 0x7f75e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:45.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.724+0000 7f75f9e06640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f75e006c8f0 0x7f75e006edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00b9f0 tx=0x7f75ec008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:45.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.727+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75e80592b0 con 0x7f75f40d3270 2024-04-15T15:16:45.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.953+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f40ba500 con 0x7f75f40d3270 2024-04-15T15:16:45.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.953+0000 7f75f27fc640 1 -- 172.21.15.46:0/1128397465 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f75e8058bb0 con 0x7f75f40d3270 2024-04-15T15:16:45.955 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:45.955 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:45.955 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f75e006c8f0 msgr2=0x7f75e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f75e006c8f0 0x7f75e006edb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00b9f0 tx=0x7f75ec008040 comp rx=0 tx=0).stop 2024-04-15T15:16:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 msgr2=0x7f75f4155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f4155220 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f75e800c4d0 tx=0x7f75e800c990 comp rx=0 tx=0).stop 2024-04-15T15:16:45.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 shutdown_connections 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f75e006c8f0 0x7f75e006edb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f75f415e700 0x7f75f40c9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75f40d3270 0x7f75f4155220 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 --2- 172.21.15.46:0/1128397465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f75f40d2360 0x7f75f40b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 >> 172.21.15.46:0/1128397465 conn(0x7f75f408bf60 msgr2=0x7f75f40d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 shutdown_connections 2024-04-15T15:16:45.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:45.955+0000 7f75fc06a640 1 -- 172.21.15.46:0/1128397465 wait complete. 2024-04-15T15:16:45.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.071+0000 7f9cde647640 1 Processor -- start 2024-04-15T15:16:46.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.071+0000 7f9cde647640 1 -- start start 2024-04-15T15:16:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cde647640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cd809b0e0 0x7f9cd8070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cde647640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cde647640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd806ff30 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cde647640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9cd8070820 con 0x7f9cd809b0e0 2024-04-15T15:16:46.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cde647640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd80cecc0 con 0x7f9cd8071d90 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cd809b0e0 0x7f9cd8070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cd809b0e0 0x7f9cd8070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41576/0 (socket says 172.21.15.46:41576) 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 -- 172.21.15.46:0/3747862394 learned_addr learned my addr 172.21.15.46:0/3747862394 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cdcbe4640 1 --2- 172.21.15.46:0/3747862394 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 -- 172.21.15.46:0/3747862394 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 msgr2=0x7f9cd80cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 --2- 172.21.15.46:0/3747862394 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 -- 172.21.15.46:0/3747862394 >> v1:172.21.15.78:6789/0 conn(0x7f9cd80ce8e0 legacy=0x7f9cd809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.072+0000 7f9cd77fe640 1 -- 172.21.15.46:0/3747862394 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd809b9d0 con 0x7f9cd809b0e0 2024-04-15T15:16:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.073+0000 7f9cd77fe640 1 --2- 172.21.15.46:0/3747862394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd809b0e0 0x7f9cd8070cb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009870 tx=0x7f9ccc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e39e0d23ac7c98a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cd6ffd640 1 -- 172.21.15.46:0/3747862394 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc0317c0 con 0x7f9cd809b0e0 2024-04-15T15:16:46.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cd6ffd640 1 -- 172.21.15.46:0/3747862394 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ccc037070 con 0x7f9cd809b0e0 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cd6ffd640 1 -- 172.21.15.46:0/3747862394 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc031b80 con 0x7f9cd809b0e0 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 -- 172.21.15.46:0/3747862394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd809b0e0 msgr2=0x7f9cd8070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 --2- 172.21.15.46:0/3747862394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd809b0e0 0x7f9cd8070cb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009870 tx=0x7f9ccc02b860 comp rx=0 tx=0).stop 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 -- 172.21.15.46:0/3747862394 shutdown_connections 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 --2- 172.21.15.46:0/3747862394 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 --2- 172.21.15.46:0/3747862394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd809b0e0 0x7f9cd8070cb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 -- 172.21.15.46:0/3747862394 >> 172.21.15.46:0/3747862394 conn(0x7f9cd80b3f40 msgr2=0x7f9cd80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 -- 172.21.15.46:0/3747862394 shutdown_connections 2024-04-15T15:16:46.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.074+0000 7f9cde647640 1 -- 172.21.15.46:0/3747862394 wait complete. 2024-04-15T15:16:46.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.075+0000 7f9cde647640 1 Processor -- start 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 -- start start 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cd809b0e0 0x7f9cd815c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 0x7f9cd815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd80704b0 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd80701d0 con 0x7f9cd809b0e0 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cde647640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd8071970 con 0x7f9cd8071d90 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cdcbe4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 0x7f9cd815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cdcbe4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 0x7f9cd815c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41582/0 (socket says 172.21.15.46:41582) 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.076+0000 7f9cdcbe4640 1 -- 172.21.15.46:0/2976197152 learned_addr learned my addr 172.21.15.46:0/2976197152 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd77fe640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cd809b0e0 0x7f9cd815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd7fff640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cdcbe4640 1 -- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 msgr2=0x7f9cd80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cdcbe4640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cdcbe4640 1 -- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cd809b0e0 msgr2=0x7f9cd815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cdcbe4640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cd809b0e0 0x7f9cd815c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cdcbe4640 1 -- 172.21.15.46:0/2976197152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd80ca840 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd77fe640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cd809b0e0 0x7f9cd815c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd7fff640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cdcbe4640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 0x7f9cd815c790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000bf70 tx=0x7f9cd0010750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0014830 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd00149d0 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd815ce30 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0014dc0 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.077+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd815d3f0 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.078+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd80689e0 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.079+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9cd001c020 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.079+0000 7f9cd4ff9640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9cc406c8f0 0x7f9cc406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.079+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9cd0090f20 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.080+0000 7f9cd7fff640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9cc406c8f0 0x7f9cc406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.080+0000 7f9cd7fff640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9cc406c8f0 0x7f9cc406edb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8003eb0 tx=0x7f9cc8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.084+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9cd80ca840 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.269+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9cd815d7f0 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.270+0000 7f9cd4ff9640 1 -- 172.21.15.46:0/2976197152 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f9cd000c040 con 0x7f9cd80ce8e0 2024-04-15T15:16:46.273 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:46.273 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:46.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9cc406c8f0 msgr2=0x7f9cc406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9cc406c8f0 0x7f9cc406edb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8003eb0 tx=0x7f9cc8009040 comp rx=0 tx=0).stop 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 msgr2=0x7f9cd815c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 0x7f9cd815c790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000bf70 tx=0x7f9cd0010750 comp rx=0 tx=0).stop 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 shutdown_connections 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9cc406c8f0 0x7f9cc406edb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cd80ce8e0 0x7f9cd815c790 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cd809b0e0 0x7f9cd815c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 --2- 172.21.15.46:0/2976197152 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9cd8071d90 0x7f9cd80cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.272+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 >> 172.21.15.46:0/2976197152 conn(0x7f9cd80b3f40 msgr2=0x7f9cd80d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.273+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 shutdown_connections 2024-04-15T15:16:46.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.273+0000 7f9cde647640 1 -- 172.21.15.46:0/2976197152 wait complete. 2024-04-15T15:16:46.289 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:36.512834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:36.868541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:40.569845+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:46.290 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.397+0000 7f72b9a88640 1 Processor -- start 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.398+0000 7f72b9a88640 1 -- start start 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.398+0000 7f72b9a88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 0x7f72b4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.398+0000 7f72b9a88640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f72b415fb30 0x7f72b40d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.398+0000 7f72b9a88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b4070720 con 0x7f72b409a9d0 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.398+0000 7f72b9a88640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f72b409ba90 con 0x7f72b415fb30 2024-04-15T15:16:46.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.398+0000 7f72b9a88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f72b40d08c0 con 0x7f72b40701f0 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b2ffd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 0x7f72b4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b3fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f72b415fb30 0x7f72b40d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b2ffd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 0x7f72b4160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:58314/0 (socket says 172.21.15.46:58314) 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b2ffd640 1 -- 172.21.15.46:0/3473357020 learned_addr learned my addr 172.21.15.46:0/3473357020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b3fff640 1 -- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 msgr2=0x7f72b4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b3fff640 1 --2- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 0x7f72b4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b3fff640 1 -- 172.21.15.46:0/3473357020 >> v1:172.21.15.78:6789/0 conn(0x7f72b409a9d0 legacy=0x7f72b409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b3fff640 1 -- 172.21.15.46:0/3473357020 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b4070a90 con 0x7f72b415fb30 2024-04-15T15:16:46.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.399+0000 7f72b2ffd640 1 --2- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 0x7f72b4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.400+0000 7f72b3fff640 1 --2- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b415fb30 0x7f72b40d0d50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009dc0 tx=0x7f72ac02be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e09fa7e3a113aace server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.400+0000 7f72b27fc640 1 -- 172.21.15.46:0/3473357020 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac039070 con 0x7f72b415fb30 2024-04-15T15:16:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.400+0000 7f72b27fc640 1 -- 172.21.15.46:0/3473357020 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72ac02cc40 con 0x7f72b415fb30 2024-04-15T15:16:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.400+0000 7f72b27fc640 1 -- 172.21.15.46:0/3473357020 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac034440 con 0x7f72b415fb30 2024-04-15T15:16:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.400+0000 7f72b9a88640 1 -- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b415fb30 msgr2=0x7f72b40d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.400+0000 7f72b9a88640 1 --2- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b415fb30 0x7f72b40d0d50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009dc0 tx=0x7f72ac02be90 comp rx=0 tx=0).stop 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.401+0000 7f72b9a88640 1 -- 172.21.15.46:0/3473357020 shutdown_connections 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.401+0000 7f72b9a88640 1 --2- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b415fb30 0x7f72b40d0d50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.401+0000 7f72b9a88640 1 --2- 172.21.15.46:0/3473357020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b40701f0 0x7f72b4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.401+0000 7f72b9a88640 1 -- 172.21.15.46:0/3473357020 >> 172.21.15.46:0/3473357020 conn(0x7f72b40d2460 msgr2=0x7f72b40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.401+0000 7f72b9a88640 1 -- 172.21.15.46:0/3473357020 shutdown_connections 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.401+0000 7f72b9a88640 1 -- 172.21.15.46:0/3473357020 wait complete. 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.402+0000 7f72b9a88640 1 Processor -- start 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.402+0000 7f72b9a88640 1 -- start start 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b40701f0 0x7f72b4157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b409a9d0 0x7f72b40c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 0x7f72b40cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72b415ffc0 con 0x7f72b415fb30 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f72b4071050 con 0x7f72b40701f0 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f72b40b1c60 con 0x7f72b409a9d0 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 0x7f72b40cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 0x7f72b40cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41598/0 (socket says 172.21.15.46:41598) 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 -- 172.21.15.46:0/1946142487 learned_addr learned my addr 172.21.15.46:0/1946142487 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b37fe640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b40701f0 0x7f72b4157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b2ffd640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b409a9d0 0x7f72b40c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 -- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b409a9d0 msgr2=0x7f72b40c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b409a9d0 0x7f72b40c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 -- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b40701f0 msgr2=0x7f72b4157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b40701f0 0x7f72b4157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 -- 172.21.15.46:0/1946142487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b40caa50 con 0x7f72b415fb30 2024-04-15T15:16:46.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b3fff640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 0x7f72b40cdae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f72ac02c4a0 tx=0x7f72ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac039050 con 0x7f72b415fb30 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72ac045070 con 0x7f72b415fb30 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b40c6e10 con 0x7f72b415fb30 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.403+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ac040690 con 0x7f72b415fb30 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.404+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b40c73d0 con 0x7f72b415fb30 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.405+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b40689e0 con 0x7f72b415fb30 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.405+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f72ac040830 con 0x7f72b415fb30 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.405+0000 7f729bfff640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72a006c8f0 0x7f72a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.405+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f72ac0ae760 con 0x7f72b415fb30 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.405+0000 7f72b37fe640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72a006c8f0 0x7f72a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.406+0000 7f72b37fe640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72a006c8f0 0x7f72a006edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f72b415f8f0 tx=0x7f72a4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.410+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72ac07c1d0 con 0x7f72b415fb30 2024-04-15T15:16:46.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.596+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72b40cec60 con 0x7f72b415fb30 2024-04-15T15:16:46.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.596+0000 7f729bfff640 1 -- 172.21.15.46:0/1946142487 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f72ac0804d0 con 0x7f72b415fb30 2024-04-15T15:16:46.598 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72a006c8f0 msgr2=0x7f72a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72a006c8f0 0x7f72a006edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f72b415f8f0 tx=0x7f72a4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 msgr2=0x7f72b40cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 0x7f72b40cdae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f72ac02c4a0 tx=0x7f72ac004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 shutdown_connections 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72a006c8f0 0x7f72a006edb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72b415fb30 0x7f72b40cdae0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72b409a9d0 0x7f72b40c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 --2- 172.21.15.46:0/1946142487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72b40701f0 0x7f72b4157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 >> 172.21.15.46:0/1946142487 conn(0x7f72b40d2460 msgr2=0x7f72b40d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 shutdown_connections 2024-04-15T15:16:46.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.598+0000 7f72b9a88640 1 -- 172.21.15.46:0/1946142487 wait complete. 2024-04-15T15:16:46.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.718+0000 7f6e9967b640 1 Processor -- start 2024-04-15T15:16:46.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.718+0000 7f6e9967b640 1 -- start start 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e9967b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6e940d5150 0x7f6e940d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e9967b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e9415eb70 con 0x7f6e940d15e0 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e9967b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e940d3c80 con 0x7f6e9415e790 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e9967b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f6e940d3790 con 0x7f6e940d5150 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e937fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6e940d5150 0x7f6e940d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e927fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6e940d15e0 0x7f6e940d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40440/0 (socket says 172.21.15.46:40440) 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e927fc640 1 -- 172.21.15.46:0/3890601079 learned_addr learned my addr 172.21.15.46:0/3890601079 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e937fe640 1 -- 172.21.15.46:0/3890601079 >> v1:172.21.15.78:6789/0 conn(0x7f6e9415e790 legacy=0x7f6e940b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e937fe640 1 -- 172.21.15.46:0/3890601079 >> v1:172.21.15.46:6789/0 conn(0x7f6e940d15e0 legacy=0x7f6e940d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e91ffb640 1 -- 172.21.15.46:0/3890601079 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 881548687 0 0) 0x7f6e9415eb70 con 0x7f6e940d15e0 2024-04-15T15:16:46.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.719+0000 7f6e937fe640 1 -- 172.21.15.46:0/3890601079 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e940d3f60 con 0x7f6e940d5150 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.720+0000 7f6e937fe640 1 --2- 172.21.15.46:0/3890601079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940d1a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6e940b61c0 tx=0x7f6e8c02bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cffcca4a7a9a409 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.720+0000 7f6e91ffb640 1 -- 172.21.15.46:0/3890601079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c039070 con 0x7f6e940d5150 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.720+0000 7f6e91ffb640 1 -- 172.21.15.46:0/3890601079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8c02cb20 con 0x7f6e940d5150 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.720+0000 7f6e91ffb640 1 -- 172.21.15.46:0/3890601079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c034440 con 0x7f6e940d5150 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 -- 172.21.15.46:0/3890601079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 msgr2=0x7f6e940d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3890601079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940d1a90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6e940b61c0 tx=0x7f6e8c02bd70 comp rx=0 tx=0).stop 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 -- 172.21.15.46:0/3890601079 shutdown_connections 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3890601079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940d1a90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 -- 172.21.15.46:0/3890601079 >> 172.21.15.46:0/3890601079 conn(0x7f6e94067b40 msgr2=0x7f6e94067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 -- 172.21.15.46:0/3890601079 shutdown_connections 2024-04-15T15:16:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.721+0000 7f6e9967b640 1 -- 172.21.15.46:0/3890601079 wait complete. 2024-04-15T15:16:46.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.722+0000 7f6e9967b640 1 Processor -- start 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.722+0000 7f6e9967b640 1 -- start start 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e9967b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6e940d15e0 0x7f6e940b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e9967b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940b8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e9967b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e9415e790 0x7f6e941f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e9967b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e940b7db0 con 0x7f6e940d5150 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e9967b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6e940d45b0 con 0x7f6e9415e790 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e9967b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6e940d29c0 con 0x7f6e940d15e0 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940b8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940b8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41624/0 (socket says 172.21.15.46:41624) 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 -- 172.21.15.46:0/3882044059 learned_addr learned my addr 172.21.15.46:0/3882044059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:46.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e937fe640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e9415e790 0x7f6e941f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e92ffd640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6e940d15e0 0x7f6e940b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 -- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6e940d15e0 msgr2=0x7f6e940b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6e940d15e0 0x7f6e940b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 -- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e9415e790 msgr2=0x7f6e941f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e9415e790 0x7f6e941f7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 -- 172.21.15.46:0/3882044059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e941f44d0 con 0x7f6e940d5150 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.723+0000 7f6e927fc640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940b8470 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00c500 tx=0x7f6e7c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.724+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c018070 con 0x7f6e940d5150 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.724+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c007580 con 0x7f6e940d5150 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.724+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c004a20 con 0x7f6e940d5150 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.724+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e941f4790 con 0x7f6e940d5150 2024-04-15T15:16:46.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.724+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e941f4d50 con 0x7f6e940d5150 2024-04-15T15:16:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.725+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e941f4f50 con 0x7f6e940d5150 2024-04-15T15:16:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.725+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e7c004bc0 con 0x7f6e940d5150 2024-04-15T15:16:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.725+0000 7f6e8b7fe640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6e7406c8f0 0x7f6e7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.726+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6e7c08b0b0 con 0x7f6e940d5150 2024-04-15T15:16:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.726+0000 7f6e92ffd640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6e7406c8f0 0x7f6e7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.726+0000 7f6e92ffd640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6e7406c8f0 0x7f6e7406edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e940b8b70 tx=0x7f6e84009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.730+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e941f4f50 con 0x7f6e940d5150 2024-04-15T15:16:46.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.917+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e940b5eb0 con 0x7f6e940d5150 2024-04-15T15:16:46.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.917+0000 7f6e8b7fe640 1 -- 172.21.15.46:0/3882044059 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f6e940b5eb0 con 0x7f6e940d5150 2024-04-15T15:16:46.919 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:46.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.919+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6e7406c8f0 msgr2=0x7f6e7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.919+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6e7406c8f0 0x7f6e7406edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6e940b8b70 tx=0x7f6e84009060 comp rx=0 tx=0).stop 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.919+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 msgr2=0x7f6e940b8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.919+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940b8470 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c00c500 tx=0x7f6e7c00c9c0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 shutdown_connections 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6e7406c8f0 0x7f6e7406edb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e9415e790 0x7f6e941f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e940d5150 0x7f6e940b8470 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 --2- 172.21.15.46:0/3882044059 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6e940d15e0 0x7f6e940b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 >> 172.21.15.46:0/3882044059 conn(0x7f6e94067b40 msgr2=0x7f6e94074ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 shutdown_connections 2024-04-15T15:16:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:46.920+0000 7f6e9967b640 1 -- 172.21.15.46:0/3882044059 wait complete. 2024-04-15T15:16:46.932 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.037+0000 7f66bdf32640 1 Processor -- start 2024-04-15T15:16:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.037+0000 7f66bdf32640 1 -- start start 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66bdf32640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66b815e700 0x7f66b80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66bdf32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b80d2360 0x7f66b80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66bdf32640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b80b47a0 con 0x7f66b815eae0 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66bdf32640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f66b80b53b0 con 0x7f66b815e700 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66bdf32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f66b80b4a80 con 0x7f66b80d2360 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66b77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66b815e700 0x7f66b80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66b77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f66b815e700 0x7f66b80b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41632/0 (socket says 172.21.15.46:41632) 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.038+0000 7f66b77fe640 1 -- 172.21.15.46:0/4265989149 learned_addr learned my addr 172.21.15.46:0/4265989149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.039+0000 7f66b6ffd640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b80d2360 0x7f66b80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.039+0000 7f66b77fe640 1 -- 172.21.15.46:0/4265989149 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b80d2360 msgr2=0x7f66b80d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.039+0000 7f66b77fe640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b80d2360 0x7f66b80d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.039+0000 7f66b77fe640 1 -- 172.21.15.46:0/4265989149 >> v1:172.21.15.78:6789/0 conn(0x7f66b815eae0 legacy=0x7f66b80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.039+0000 7f66b77fe640 1 -- 172.21.15.46:0/4265989149 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b80b4d60 con 0x7f66b815e700 2024-04-15T15:16:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.039+0000 7f66b6ffd640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b80d2360 0x7f66b80d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66b77fe640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b815e700 0x7f66b80b5840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009ea0 tx=0x7f66a802bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79afd05624ac435 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66b67fc640 1 -- 172.21.15.46:0/4265989149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a802cab0 con 0x7f66b815e700 2024-04-15T15:16:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66b67fc640 1 -- 172.21.15.46:0/4265989149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a802cc50 con 0x7f66b815e700 2024-04-15T15:16:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66b67fc640 1 -- 172.21.15.46:0/4265989149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80343f0 con 0x7f66b815e700 2024-04-15T15:16:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 -- 172.21.15.46:0/4265989149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b815e700 msgr2=0x7f66b80b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b815e700 0x7f66b80b5840 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009ea0 tx=0x7f66a802bf10 comp rx=0 tx=0).stop 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 -- 172.21.15.46:0/4265989149 shutdown_connections 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b80d2360 0x7f66b80d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 --2- 172.21.15.46:0/4265989149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b815e700 0x7f66b80b5840 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 -- 172.21.15.46:0/4265989149 >> 172.21.15.46:0/4265989149 conn(0x7f66b808bf60 msgr2=0x7f66b80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 -- 172.21.15.46:0/4265989149 shutdown_connections 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.040+0000 7f66bdf32640 1 -- 172.21.15.46:0/4265989149 wait complete. 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.041+0000 7f66bdf32640 1 Processor -- start 2024-04-15T15:16:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 -- start start 2024-04-15T15:16:47.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 0x7f66b80caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b815e700 0x7f66b80b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b815eae0 0x7f66b80b92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80b5d80 con 0x7f66b80d2360 2024-04-15T15:16:47.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80b4450 con 0x7f66b815e700 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66bdf32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f66b80d1d40 con 0x7f66b815eae0 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.042+0000 7f66b77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 0x7f66b80caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 0x7f66b80caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41634/0 (socket says 172.21.15.46:41634) 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 -- 172.21.15.46:0/2617234833 learned_addr learned my addr 172.21.15.46:0/2617234833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 -- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b815eae0 msgr2=0x7f66b80b92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b6ffd640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b815e700 0x7f66b80b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b7fff640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b815eae0 0x7f66b80b92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b815eae0 0x7f66b80b92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 -- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b815e700 msgr2=0x7f66b80b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b815e700 0x7f66b80b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 -- 172.21.15.46:0/2617234833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b80ce440 con 0x7f66b80d2360 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b6ffd640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b815e700 0x7f66b80b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b7fff640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b815eae0 0x7f66b80b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66b77fe640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 0x7f66b80caa10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009ea0 tx=0x7f66a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8043020 con 0x7f66b80d2360 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a803e450 con 0x7f66b80d2360 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b80ce720 con 0x7f66b80d2360 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8009510 con 0x7f66b80d2360 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.043+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b80b99c0 con 0x7f66b80d2360 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.044+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b80689e0 con 0x7f66b80d2360 2024-04-15T15:16:47.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.045+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f66a8046020 con 0x7f66b80d2360 2024-04-15T15:16:47.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.045+0000 7f669ffff640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f66a406c8f0 0x7f66a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.045+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f66a80ae710 con 0x7f66b80d2360 2024-04-15T15:16:47.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.045+0000 7f66b6ffd640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f66a406c8f0 0x7f66a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.046+0000 7f66b6ffd640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f66a406c8f0 0x7f66a406edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f66ac006fd0 tx=0x7f66ac008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.049+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66a804d030 con 0x7f66b80d2360 2024-04-15T15:16:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.235+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f66b80ba500 con 0x7f66b80d2360 2024-04-15T15:16:47.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.235+0000 7f669ffff640 1 -- 172.21.15.46:0/2617234833 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f66a807c740 con 0x7f66b80d2360 2024-04-15T15:16:47.237 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f66a406c8f0 msgr2=0x7f66a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f66a406c8f0 0x7f66a406edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f66ac006fd0 tx=0x7f66ac008040 comp rx=0 tx=0).stop 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 msgr2=0x7f66b80caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 0x7f66b80caa10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009ea0 tx=0x7f66a8004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 shutdown_connections 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f66a406c8f0 0x7f66a406edb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b815eae0 0x7f66b80b92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b815e700 0x7f66b80b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 --2- 172.21.15.46:0/2617234833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b80d2360 0x7f66b80caa10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 >> 172.21.15.46:0/2617234833 conn(0x7f66b808bf60 msgr2=0x7f66b80d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 shutdown_connections 2024-04-15T15:16:47.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.237+0000 7f66bdf32640 1 -- 172.21.15.46:0/2617234833 wait complete. 2024-04-15T15:16:47.250 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.356+0000 7fa88df59640 1 Processor -- start 2024-04-15T15:16:47.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.356+0000 7fa88df59640 1 -- start start 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa88df59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa88815e790 0x7fa8880d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa88df59640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa8880cfd20 0x7fa8880d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa88df59640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8880d0f00 0x7fa8880d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa88df59640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa8880d2db0 con 0x7fa8880d0f00 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa88df59640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa8880d2480 con 0x7fa8880cfd20 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa88df59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa8880d2760 con 0x7fa88815e790 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8880d0f00 0x7fa8880d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8880d0f00 0x7fa8880d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41636/0 (socket says 172.21.15.46:41636) 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 -- 172.21.15.46:0/1778313460 learned_addr learned my addr 172.21.15.46:0/1778313460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:47.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa8877fe640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa88815e790 0x7fa8880d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 -- 172.21.15.46:0/1778313460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa88815e790 msgr2=0x7fa8880d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa886ffd640 1 --2- 172.21.15.46:0/1778313460 >> v2:172.21.15.78:3300/0 conn(0x7fa8880cfd20 0x7fa8880d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa88815e790 0x7fa8880d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 -- 172.21.15.46:0/1778313460 >> v2:172.21.15.78:3300/0 conn(0x7fa8880cfd20 msgr2=0x7fa8880d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 --2- 172.21.15.46:0/1778313460 >> v2:172.21.15.78:3300/0 conn(0x7fa8880cfd20 0x7fa8880d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa887fff640 1 -- 172.21.15.46:0/1778313460 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8880d2a40 con 0x7fa8880d0f00 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.357+0000 7fa8877fe640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa88815e790 0x7fa8880d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.358+0000 7fa887fff640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8880d0f00 0x7fa8880d13b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa88808d020 tx=0x7fa88002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2ade5f28308d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.358+0000 7fa8867fc640 1 -- 172.21.15.46:0/1778313460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa88002ca70 con 0x7fa8880d0f00 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.358+0000 7fa8867fc640 1 -- 172.21.15.46:0/1778313460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa88002cc10 con 0x7fa8880d0f00 2024-04-15T15:16:47.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.358+0000 7fa8867fc640 1 -- 172.21.15.46:0/1778313460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8800343f0 con 0x7fa8880d0f00 2024-04-15T15:16:47.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 -- 172.21.15.46:0/1778313460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8880d0f00 msgr2=0x7fa8880d13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8880d0f00 0x7fa8880d13b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa88808d020 tx=0x7fa88002bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 -- 172.21.15.46:0/1778313460 shutdown_connections 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8880d0f00 0x7fa8880d13b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 --2- 172.21.15.46:0/1778313460 >> v2:172.21.15.78:3300/0 conn(0x7fa8880cfd20 0x7fa8880d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 --2- 172.21.15.46:0/1778313460 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa88815e790 0x7fa8880d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 -- 172.21.15.46:0/1778313460 >> 172.21.15.46:0/1778313460 conn(0x7fa88808bf60 msgr2=0x7fa8880d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 -- 172.21.15.46:0/1778313460 shutdown_connections 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.359+0000 7fa88df59640 1 -- 172.21.15.46:0/1778313460 wait complete. 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.360+0000 7fa88df59640 1 Processor -- start 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.360+0000 7fa88df59640 1 -- start start 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa88df59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8880cfd20 0x7fa8881f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa88df59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8880d0f00 0x7fa8880c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa88df59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 0x7fa888159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa88df59640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa88815ecf0 con 0x7fa88815e790 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa88df59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa88815eb70 con 0x7fa8880cfd20 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa88df59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa8880b74f0 con 0x7fa8880d0f00 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 0x7fa888159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 0x7fa888159dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41638/0 (socket says 172.21.15.46:41638) 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 -- 172.21.15.46:0/850089334 learned_addr learned my addr 172.21.15.46:0/850089334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa886ffd640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8880d0f00 0x7fa8880c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa8877fe640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8880cfd20 0x7fa8881f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 -- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8880d0f00 msgr2=0x7fa8880c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8880d0f00 0x7fa8880c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 -- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8880cfd20 msgr2=0x7fa8881f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8880cfd20 0x7fa8881f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa887fff640 1 -- 172.21.15.46:0/850089334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88815a8e0 con 0x7fa88815e790 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.361+0000 7fa8877fe640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8880cfd20 0x7fa8881f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.362+0000 7fa887fff640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 0x7fa888159dd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa880036040 tx=0x7fa880004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.362+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880043020 con 0x7fa88815e790 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.362+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa88003e450 con 0x7fa88815e790 2024-04-15T15:16:47.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.362+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880009510 con 0x7fa88815e790 2024-04-15T15:16:47.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.362+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8880cb810 con 0x7fa88815e790 2024-04-15T15:16:47.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.362+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8880cbdd0 con 0x7fa88815e790 2024-04-15T15:16:47.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.363+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8880689e0 con 0x7fa88815e790 2024-04-15T15:16:47.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.364+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa880046020 con 0x7fa88815e790 2024-04-15T15:16:47.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.364+0000 7fa86ffff640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa87406c8f0 0x7fa87406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.364+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa8800ae740 con 0x7fa88815e790 2024-04-15T15:16:47.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.364+0000 7fa8877fe640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa87406c8f0 0x7fa87406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.365+0000 7fa8877fe640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa87406c8f0 0x7fa87406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa87800a990 tx=0x7fa878008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.368+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa88004d030 con 0x7fa88815e790 2024-04-15T15:16:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.552+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa8880d1ce0 con 0x7fa88815e790 2024-04-15T15:16:47.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.552+0000 7fa86ffff640 1 -- 172.21.15.46:0/850089334 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fa88007c800 con 0x7fa88815e790 2024-04-15T15:16:47.554 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:16:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa87406c8f0 msgr2=0x7fa87406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa87406c8f0 0x7fa87406edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa87800a990 tx=0x7fa878008040 comp rx=0 tx=0).stop 2024-04-15T15:16:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 msgr2=0x7fa888159dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 0x7fa888159dd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa880036040 tx=0x7fa880004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 shutdown_connections 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa87406c8f0 0x7fa87406edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa88815e790 0x7fa888159dd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa8880d0f00 0x7fa8880c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 --2- 172.21.15.46:0/850089334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8880cfd20 0x7fa8881f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 >> 172.21.15.46:0/850089334 conn(0x7fa88808bf60 msgr2=0x7fa88806e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 shutdown_connections 2024-04-15T15:16:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.554+0000 7fa88df59640 1 -- 172.21.15.46:0/850089334 wait complete. 2024-04-15T15:16:47.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:47.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.669+0000 7f8c52893640 1 Processor -- start 2024-04-15T15:16:47.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 -- start start 2024-04-15T15:16:47.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8c4c070940 0x7f8c4c09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4c0d0da0 0x7f8c4c0d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8c4c0d05e0 con 0x7f8c4c0d0da0 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f8c4c071260 con 0x7f8c4c070940 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c52893640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c15f720 con 0x7f8c4c160310 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.670+0000 7f8c50e30640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4c0d0da0 0x7f8c4c0d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4c0d0da0 0x7f8c4c0d1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41646/0 (socket says 172.21.15.46:41646) 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 -- 172.21.15.46:0/3599493664 learned_addr learned my addr 172.21.15.46:0/3599493664 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c43fff640 1 --2- 172.21.15.46:0/3599493664 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/3599493664 >> v2:172.21.15.78:3300/0 conn(0x7f8c4c070940 0x7f8c4c09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 -- 172.21.15.46:0/3599493664 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 msgr2=0x7f8c4c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 --2- 172.21.15.46:0/3599493664 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 -- 172.21.15.46:0/3599493664 >> v2:172.21.15.78:3300/0 conn(0x7f8c4c070940 msgr2=0x7f8c4c09b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 --2- 172.21.15.46:0/3599493664 >> v2:172.21.15.78:3300/0 conn(0x7f8c4c070940 0x7f8c4c09b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c50e30640 1 -- 172.21.15.46:0/3599493664 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c15fa00 con 0x7f8c4c0d0da0 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.671+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/3599493664 >> v2:172.21.15.78:3300/0 conn(0x7f8c4c070940 0x7f8c4c09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c50e30640 1 --2- 172.21.15.46:0/3599493664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c0d1230 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0741c0 tx=0x7f8c4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f707d3d86cd829 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c4b7fe640 1 -- 172.21.15.46:0/3599493664 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4402ca70 con 0x7f8c4c0d0da0 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c4b7fe640 1 -- 172.21.15.46:0/3599493664 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c4402cc10 con 0x7f8c4c0d0da0 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c4b7fe640 1 -- 172.21.15.46:0/3599493664 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c440343f0 con 0x7f8c4c0d0da0 2024-04-15T15:16:47.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 -- 172.21.15.46:0/3599493664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d0da0 msgr2=0x7f8c4c0d1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 --2- 172.21.15.46:0/3599493664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c0d1230 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0741c0 tx=0x7f8c4402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 -- 172.21.15.46:0/3599493664 shutdown_connections 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 --2- 172.21.15.46:0/3599493664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c0d1230 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 --2- 172.21.15.46:0/3599493664 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 --2- 172.21.15.46:0/3599493664 >> v2:172.21.15.78:3300/0 conn(0x7f8c4c070940 0x7f8c4c09b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 -- 172.21.15.46:0/3599493664 >> 172.21.15.46:0/3599493664 conn(0x7f8c4c0b3f40 msgr2=0x7f8c4c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 -- 172.21.15.46:0/3599493664 shutdown_connections 2024-04-15T15:16:47.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.672+0000 7f8c52893640 1 -- 172.21.15.46:0/3599493664 wait complete. 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 Processor -- start 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 -- start start 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 0x7f8c4c15d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c15d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c160770 con 0x7f8c4c070940 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c0d0880 con 0x7f8c4c0d0da0 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.674+0000 7f8c52893640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c0d1770 con 0x7f8c4c160310 2024-04-15T15:16:47.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 0x7f8c4c15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 0x7f8c4c15d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41660/0 (socket says 172.21.15.46:41660) 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 -- 172.21.15.46:0/937298064 learned_addr learned my addr 172.21.15.46:0/937298064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c43fff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c15d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c50e30640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 -- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 msgr2=0x7f8c4c15c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c15c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 -- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c0d0da0 msgr2=0x7f8c4c15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c15d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 -- 172.21.15.46:0/937298064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c0c84f0 con 0x7f8c4c070940 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c43fff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c50e30640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c4bfff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 0x7f8c4c15d280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00e840 tx=0x7f8c3c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c00f040 con 0x7f8c4c070940 2024-04-15T15:16:47.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3c004030 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c004400 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c0c87d0 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.675+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c0c8d90 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.676+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c0689e0 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.677+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c3c00c6d0 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.677+0000 7f8c497fa640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.677+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8c3c08bcb0 con 0x7f8c4c070940 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.677+0000 7f8c43fff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.678+0000 7f8c43fff640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0b7440 tx=0x7f8c34008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.681+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c3c017020 con 0x7f8c4c070940 2024-04-15T15:16:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.906+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c4c071a90 con 0x7f8c4c070940 2024-04-15T15:16:47.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.906+0000 7f8c497fa640 1 -- 172.21.15.46:0/937298064 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f8c3c059c60 con 0x7f8c4c070940 2024-04-15T15:16:47.908 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:47.908 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:47.909 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:47.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 msgr2=0x7f8c2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c0b7440 tx=0x7f8c34008040 comp rx=0 tx=0).stop 2024-04-15T15:16:47.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 msgr2=0x7f8c4c15d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 0x7f8c4c15d280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00e840 tx=0x7f8c3c00ed00 comp rx=0 tx=0).stop 2024-04-15T15:16:47.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 shutdown_connections 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4c160310 0x7f8c4c15c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c4c0d0da0 0x7f8c4c15d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 --2- 172.21.15.46:0/937298064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4c070940 0x7f8c4c15d280 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 >> 172.21.15.46:0/937298064 conn(0x7f8c4c0b3f40 msgr2=0x7f8c4c0b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 shutdown_connections 2024-04-15T15:16:47.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:47.908+0000 7f8c52893640 1 -- 172.21.15.46:0/937298064 wait complete. 2024-04-15T15:16:47.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:48.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.025+0000 7f3312175640 1 Processor -- start 2024-04-15T15:16:48.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.025+0000 7f3312175640 1 -- start start 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.025+0000 7f3312175640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f330c15e790 0x7f330c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.025+0000 7f3312175640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c0cfd20 0x7f330c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.025+0000 7f3312175640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330c0d2480 con 0x7f330c0d0f00 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f3312175640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f330c0d2db0 con 0x7f330c15e790 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f3312175640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f330c0d2760 con 0x7f330c0cfd20 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330affd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c0cfd20 0x7f330c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330b7fe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f330c15e790 0x7f330c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f330c0d0f00 0x7f330c0b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:51376/0 (socket says 172.21.15.46:51376) 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330bfff640 1 -- 172.21.15.46:0/2480169270 learned_addr learned my addr 172.21.15.46:0/2480169270 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:48.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330b7fe640 1 -- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c0cfd20 msgr2=0x7f330c0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330b7fe640 1 --2- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c0cfd20 0x7f330c0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330b7fe640 1 -- 172.21.15.46:0/2480169270 >> v1:172.21.15.78:6789/0 conn(0x7f330c0d0f00 legacy=0x7f330c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330b7fe640 1 -- 172.21.15.46:0/2480169270 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c0d2a40 con 0x7f330c15e790 2024-04-15T15:16:48.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.026+0000 7f330affd640 1 --2- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c0cfd20 0x7f330c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:48.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.027+0000 7f330b7fe640 1 --2- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c15e790 0x7f330c0d3240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0097a0 tx=0x7f32fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cefb0272c6d2f30b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.027+0000 7f330a7fc640 1 -- 172.21.15.46:0/2480169270 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc035020 con 0x7f330c15e790 2024-04-15T15:16:48.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.027+0000 7f330a7fc640 1 -- 172.21.15.46:0/2480169270 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32fc031830 con 0x7f330c15e790 2024-04-15T15:16:48.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.027+0000 7f330a7fc640 1 -- 172.21.15.46:0/2480169270 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32fc031c00 con 0x7f330c15e790 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 -- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c15e790 msgr2=0x7f330c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 --2- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c15e790 0x7f330c0d3240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0097a0 tx=0x7f32fc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 -- 172.21.15.46:0/2480169270 shutdown_connections 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 --2- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c0cfd20 0x7f330c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 --2- 172.21.15.46:0/2480169270 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c15e790 0x7f330c0d3240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 -- 172.21.15.46:0/2480169270 >> 172.21.15.46:0/2480169270 conn(0x7f330c08bf60 msgr2=0x7f330c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 -- 172.21.15.46:0/2480169270 shutdown_connections 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.028+0000 7f3312175640 1 -- 172.21.15.46:0/2480169270 wait complete. 2024-04-15T15:16:48.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.029+0000 7f3312175640 1 Processor -- start 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.029+0000 7f3312175640 1 -- start start 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f3312175640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c0cfd20 0x7f330c0ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f3312175640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 0x7f330c0c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f3312175640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c15e790 0x7f330c1e9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f3312175640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f330c15ecf0 con 0x7f330c0d0f00 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f3312175640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f330c15eb70 con 0x7f330c0cfd20 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f3312175640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f330c0d13b0 con 0x7f330c15e790 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 0x7f330c0c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 0x7f330c0c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41672/0 (socket says 172.21.15.46:41672) 2024-04-15T15:16:48.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 -- 172.21.15.46:0/2292080383 learned_addr learned my addr 172.21.15.46:0/2292080383 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330bfff640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c15e790 0x7f330c1e9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330b7fe640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c0cfd20 0x7f330c0ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 -- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c15e790 msgr2=0x7f330c1e9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c15e790 0x7f330c1e9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 -- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c0cfd20 msgr2=0x7f330c0ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c0cfd20 0x7f330c0ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 -- 172.21.15.46:0/2292080383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330c0c8f90 con 0x7f330c0d0f00 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330bfff640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c15e790 0x7f330c1e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.030+0000 7f330affd640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 0x7f330c0c8740 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f330000ebe0 tx=0x7f330000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.031+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300017070 con 0x7f330c0d0f00 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.031+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f330000cc30 con 0x7f330c0d0f00 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.031+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330c0c9270 con 0x7f330c0d0f00 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.031+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300013470 con 0x7f330c0d0f00 2024-04-15T15:16:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.031+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330c1f5a50 con 0x7f330c0d0f00 2024-04-15T15:16:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.032+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f330c0c9790 con 0x7f330c0d0f00 2024-04-15T15:16:48.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.032+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f330001a070 con 0x7f330c0d0f00 2024-04-15T15:16:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.033+0000 7f32f3fff640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f806c8f0 0x7f32f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.033+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f330008c1e0 con 0x7f330c0d0f00 2024-04-15T15:16:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.033+0000 7f330b7fe640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f806c8f0 0x7f32f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.033+0000 7f330b7fe640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f806c8f0 0x7f32f806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f32fc036000 tx=0x7f32fc03c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.037+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f330c0c9790 con 0x7f330c0d0f00 2024-04-15T15:16:48.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.222+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f330c0689e0 con 0x7f330c0d0f00 2024-04-15T15:16:48.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.223+0000 7f32f3fff640 1 -- 172.21.15.46:0/2292080383 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f330005a2a0 con 0x7f330c0d0f00 2024-04-15T15:16:48.226 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:48.226 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f806c8f0 msgr2=0x7f32f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f806c8f0 0x7f32f806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f32fc036000 tx=0x7f32fc03c040 comp rx=0 tx=0).stop 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 msgr2=0x7f330c0c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 0x7f330c0c8740 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f330000ebe0 tx=0x7f330000c520 comp rx=0 tx=0).stop 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 shutdown_connections 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32f806c8f0 0x7f32f806edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f330c15e790 0x7f330c1e9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f330c0d0f00 0x7f330c0c8740 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 --2- 172.21.15.46:0/2292080383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f330c0cfd20 0x7f330c0ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.225+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 >> 172.21.15.46:0/2292080383 conn(0x7f330c08bf60 msgr2=0x7f330c0caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.226+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 shutdown_connections 2024-04-15T15:16:48.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.226+0000 7f3312175640 1 -- 172.21.15.46:0/2292080383 wait complete. 2024-04-15T15:16:48.239 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:16:48.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.342+0000 7fac657d4640 1 Processor -- start 2024-04-15T15:16:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.342+0000 7fac657d4640 1 -- start start 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.342+0000 7fac657d4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac600af340 0x7fac600b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac657d4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac600b1620 0x7fac600b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac657d4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac600af920 con 0x7fac6015ecb0 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac657d4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fac6015f090 con 0x7fac600af340 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac657d4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fac600cdf80 con 0x7fac600b1620 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac567fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac600af340 0x7fac600b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac5effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fac6015ecb0 0x7fac600b0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40456/0 (socket says 172.21.15.46:40456) 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac5effd640 1 -- 172.21.15.46:0/530345842 learned_addr learned my addr 172.21.15.46:0/530345842 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac567fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fac600af340 0x7fac600b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41684/0 (socket says 172.21.15.46:41684) 2024-04-15T15:16:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac5f7fe640 1 --2- 172.21.15.46:0/530345842 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac600b1620 0x7fac600b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac567fc640 1 -- 172.21.15.46:0/530345842 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac600b1620 msgr2=0x7fac600b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac567fc640 1 --2- 172.21.15.46:0/530345842 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac600b1620 0x7fac600b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac567fc640 1 -- 172.21.15.46:0/530345842 >> v1:172.21.15.46:6789/0 conn(0x7fac6015ecb0 legacy=0x7fac600b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac567fc640 1 -- 172.21.15.46:0/530345842 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac600afc00 con 0x7fac600af340 2024-04-15T15:16:48.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.343+0000 7fac5e7fc640 1 -- 172.21.15.46:0/530345842 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3430892138 0 0) 0x7fac600af920 con 0x7fac6015ecb0 2024-04-15T15:16:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.344+0000 7fac567fc640 1 --2- 172.21.15.46:0/530345842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600af340 0x7fac600b4350 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fac480097a0 tx=0x7fac4802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b58ae5728da3e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.344+0000 7fac5e7fc640 1 -- 172.21.15.46:0/530345842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48034070 con 0x7fac600af340 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.344+0000 7fac5e7fc640 1 -- 172.21.15.46:0/530345842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac48030610 con 0x7fac600af340 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.344+0000 7fac5e7fc640 1 -- 172.21.15.46:0/530345842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480309e0 con 0x7fac600af340 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.344+0000 7fac657d4640 1 -- 172.21.15.46:0/530345842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600af340 msgr2=0x7fac600b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.344+0000 7fac657d4640 1 --2- 172.21.15.46:0/530345842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600af340 0x7fac600b4350 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fac480097a0 tx=0x7fac4802b960 comp rx=0 tx=0).stop 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.345+0000 7fac657d4640 1 -- 172.21.15.46:0/530345842 shutdown_connections 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.345+0000 7fac657d4640 1 --2- 172.21.15.46:0/530345842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600af340 0x7fac600b4350 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.345+0000 7fac657d4640 1 --2- 172.21.15.46:0/530345842 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac600b1620 0x7fac600b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.345+0000 7fac657d4640 1 -- 172.21.15.46:0/530345842 >> 172.21.15.46:0/530345842 conn(0x7fac600b6870 msgr2=0x7fac600b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.345+0000 7fac657d4640 1 -- 172.21.15.46:0/530345842 shutdown_connections 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.345+0000 7fac657d4640 1 -- 172.21.15.46:0/530345842 wait complete. 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.346+0000 7fac657d4640 1 Processor -- start 2024-04-15T15:16:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.346+0000 7fac657d4640 1 -- start start 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac600af340 0x7fac6014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 0x7fac60160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac6015ecb0 0x7fac601607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fac600ce410 con 0x7fac600b1620 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fac600b4890 con 0x7fac600af340 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fac600ce590 con 0x7fac6015ecb0 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 0x7fac60160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 0x7fac60160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41692/0 (socket says 172.21.15.46:41692) 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 -- 172.21.15.46:0/40283112 learned_addr learned my addr 172.21.15.46:0/40283112 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac5f7fe640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac6015ecb0 0x7fac601607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac5effd640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac600af340 0x7fac6014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 -- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac6015ecb0 msgr2=0x7fac601607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac6015ecb0 0x7fac601607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 -- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac600af340 msgr2=0x7fac6014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac600af340 0x7fac6014ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 -- 172.21.15.46:0/40283112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6009b750 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac567fc640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 0x7fac60160260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fac4802bf70 tx=0x7fac480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48034040 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac48030410 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac6014d730 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48039c30 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.347+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6015c850 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.348+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac600b5a10 con 0x7fac600b1620 2024-04-15T15:16:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.349+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac48037070 con 0x7fac600b1620 2024-04-15T15:16:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.349+0000 7fac57fff640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac4006c8f0 0x7fac4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.349+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fac480ade60 con 0x7fac600b1620 2024-04-15T15:16:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.349+0000 7fac5effd640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac4006c8f0 0x7fac4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.350+0000 7fac5effd640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac4006c8f0 0x7fac4006edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fac50000f60 tx=0x7fac50007c00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.353+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac600b5a10 con 0x7fac600b1620 2024-04-15T15:16:48.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.580+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac600689e0 con 0x7fac600b1620 2024-04-15T15:16:48.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.580+0000 7fac57fff640 1 -- 172.21.15.46:0/40283112 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7fac4807b9f0 con 0x7fac600b1620 2024-04-15T15:16:48.582 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:48.582 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:48.583 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:48.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac4006c8f0 msgr2=0x7fac4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac4006c8f0 0x7fac4006edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fac50000f60 tx=0x7fac50007c00 comp rx=0 tx=0).stop 2024-04-15T15:16:48.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 msgr2=0x7fac60160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 0x7fac60160260 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fac4802bf70 tx=0x7fac480046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 shutdown_connections 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fac4006c8f0 0x7fac4006edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fac6015ecb0 0x7fac601607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac600b1620 0x7fac60160260 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 --2- 172.21.15.46:0/40283112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac600af340 0x7fac6014ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 >> 172.21.15.46:0/40283112 conn(0x7fac600b6870 msgr2=0x7fac6015de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 shutdown_connections 2024-04-15T15:16:48.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.582+0000 7fac657d4640 1 -- 172.21.15.46:0/40283112 wait complete. 2024-04-15T15:16:48.596 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.699+0000 7f0f57e4a640 1 Processor -- start 2024-04-15T15:16:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.699+0000 7f0f57e4a640 1 -- start start 2024-04-15T15:16:48.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f57e4a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f500d3240 0x7f0f500d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f57e4a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0f500d1440 0x7f0f500b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f57e4a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f500d2480 con 0x7f0f5015e790 2024-04-15T15:16:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f57e4a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0f500d2db0 con 0x7f0f500d3240 2024-04-15T15:16:48.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f57e4a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0f500d2760 con 0x7f0f500d1440 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f553e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0f500d3240 0x7f0f500d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f55be6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0f5015e790 0x7f0f500cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40468/0 (socket says 172.21.15.46:40468) 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.700+0000 7f0f55be6640 1 -- 172.21.15.46:0/894148696 learned_addr learned my addr 172.21.15.46:0/894148696 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f553e5640 1 -- 172.21.15.46:0/894148696 >> v2:172.21.15.78:3300/0 conn(0x7f0f500d1440 msgr2=0x7f0f500b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f553e5640 1 --2- 172.21.15.46:0/894148696 >> v2:172.21.15.78:3300/0 conn(0x7f0f500d1440 0x7f0f500b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f553e5640 1 -- 172.21.15.46:0/894148696 >> v1:172.21.15.46:6789/0 conn(0x7f0f5015e790 legacy=0x7f0f500cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f553e5640 1 -- 172.21.15.46:0/894148696 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f500d2a40 con 0x7f0f500d3240 2024-04-15T15:16:48.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f54be4640 1 -- 172.21.15.46:0/894148696 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4237079008 0 0) 0x7f0f500d2480 con 0x7f0f5015e790 2024-04-15T15:16:48.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f553e5640 1 --2- 172.21.15.46:0/894148696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f500d3240 0x7f0f500d0f00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0f500c8120 tx=0x7f0f4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bf0c93ceaacaab server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.701+0000 7f0f54be4640 1 -- 172.21.15.46:0/894148696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f40035020 con 0x7f0f500d3240 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f54be4640 1 -- 172.21.15.46:0/894148696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f40031830 con 0x7f0f500d3240 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f54be4640 1 -- 172.21.15.46:0/894148696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f40031c00 con 0x7f0f500d3240 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 -- 172.21.15.46:0/894148696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f500d3240 msgr2=0x7f0f500d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/894148696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f500d3240 0x7f0f500d0f00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0f500c8120 tx=0x7f0f4002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 -- 172.21.15.46:0/894148696 shutdown_connections 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/894148696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f500d3240 0x7f0f500d0f00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/894148696 >> v2:172.21.15.78:3300/0 conn(0x7f0f500d1440 0x7f0f500b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 -- 172.21.15.46:0/894148696 >> 172.21.15.46:0/894148696 conn(0x7f0f5008bf60 msgr2=0x7f0f500d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 -- 172.21.15.46:0/894148696 shutdown_connections 2024-04-15T15:16:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.702+0000 7f0f57e4a640 1 -- 172.21.15.46:0/894148696 wait complete. 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.703+0000 7f0f57e4a640 1 Processor -- start 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.703+0000 7f0f57e4a640 1 -- start start 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f57e4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f500d1440 0x7f0f50151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f57e4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f500d3240 0x7f0f5015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f57e4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 0x7f0f501f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f57e4a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0f500d5180 con 0x7f0f5015e790 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f57e4a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0f500d0430 con 0x7f0f500d1440 2024-04-15T15:16:48.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f57e4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0f500d5300 con 0x7f0f500d3240 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f563e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 0x7f0f501f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f563e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 0x7f0f501f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41702/0 (socket says 172.21.15.46:41702) 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f563e7640 1 -- 172.21.15.46:0/2619264924 learned_addr learned my addr 172.21.15.46:0/2619264924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f553e5640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f500d3240 0x7f0f5015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.704+0000 7f0f55be6640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f500d1440 0x7f0f50151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f563e7640 1 -- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f500d3240 msgr2=0x7f0f5015b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f563e7640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f500d3240 0x7f0f5015b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f563e7640 1 -- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f500d1440 msgr2=0x7f0f50151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f563e7640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f500d1440 0x7f0f50151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f563e7640 1 -- 172.21.15.46:0/2619264924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f500cd0b0 con 0x7f0f5015e790 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f563e7640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 0x7f0f501f4060 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800b970 tx=0x7f0f4800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48019070 con 0x7f0f5015e790 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f4800cb60 con 0x7f0f5015e790 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f500cd390 con 0x7f0f5015e790 2024-04-15T15:16:48.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48014440 con 0x7f0f5015e790 2024-04-15T15:16:48.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.705+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f500cd950 con 0x7f0f5015e790 2024-04-15T15:16:48.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.706+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f500689e0 con 0x7f0f5015e790 2024-04-15T15:16:48.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.707+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f4801c070 con 0x7f0f5015e790 2024-04-15T15:16:48.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.707+0000 7f0f467fc640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f3406c8f0 0x7f0f3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.707+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0f4808bca0 con 0x7f0f5015e790 2024-04-15T15:16:48.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.707+0000 7f0f55be6640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f3406c8f0 0x7f0f3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.708+0000 7f0f55be6640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f3406c8f0 0x7f0f3406edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c007fd0 tx=0x7f0f4c007de0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.711+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f480595f0 con 0x7f0f5015e790 2024-04-15T15:16:48.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.934+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f500d1ce0 con 0x7f0f5015e790 2024-04-15T15:16:48.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.935+0000 7f0f467fc640 1 -- 172.21.15.46:0/2619264924 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f0f4805d8f0 con 0x7f0f5015e790 2024-04-15T15:16:48.937 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:48.937 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/226905862","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":226905862},{"type":"v1","addr":"172.21.15.46:6834","nonce":226905862}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4266,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2943834711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2943834711},{"type":"v1","addr":"172.21.15.46:6835","nonce":2943834711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/1379923145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1379923145},{"type":"v1","addr":"172.21.15.78:6836","nonce":1379923145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:37.926112+0000","modified":"2024-04-15T15:16:00.995766+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4284},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4284":{"gid":4284,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.78:6837/3854702652","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":3854702652},{"type":"v1","addr":"172.21.15.78:6837","nonce":3854702652}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4284,"qdb_cluster":[4284]},"id":1}]} 2024-04-15T15:16:48.937 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f3406c8f0 msgr2=0x7f0f3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f3406c8f0 0x7f0f3406edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c007fd0 tx=0x7f0f4c007de0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 msgr2=0x7f0f501f4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 0x7f0f501f4060 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800b970 tx=0x7f0f4800be30 comp rx=0 tx=0).stop 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 shutdown_connections 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0f3406c8f0 0x7f0f3406edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f5015e790 0x7f0f501f4060 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0f500d3240 0x7f0f5015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 --2- 172.21.15.46:0/2619264924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f500d1440 0x7f0f50151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 >> 172.21.15.46:0/2619264924 conn(0x7f0f5008bf60 msgr2=0x7f0f500d4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 shutdown_connections 2024-04-15T15:16:48.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:48.937+0000 7f0f57e4a640 1 -- 172.21.15.46:0/2619264924 wait complete. 2024-04-15T15:16:48.950 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:49.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.053+0000 7f64a4835640 1 Processor -- start 2024-04-15T15:16:49.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.053+0000 7f64a4835640 1 -- start start 2024-04-15T15:16:49.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f64a4835640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 0x7f64a00d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f64a4835640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a00d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f64a4835640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a00d2480 con 0x7f64a00d0f00 2024-04-15T15:16:49.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f64a4835640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f64a00d2db0 con 0x7f64a015e790 2024-04-15T15:16:49.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f64a4835640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f64a00d2760 con 0x7f64a00cfd20 2024-04-15T15:16:49.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f649e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 0x7f64a00d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.054+0000 7f649dd9b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a00d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649e59c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 0x7f64a00d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:60006/0 (socket says 172.21.15.46:60006) 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649dd9b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a00d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:58438/0 (socket says 172.21.15.46:58438) 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f64a00d0f00 0x7f64a00b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:51382/0 (socket says 172.21.15.46:51382) 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649e59c640 1 -- 172.21.15.46:0/190518569 learned_addr learned my addr 172.21.15.46:0/190518569 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649dd9b640 1 -- 172.21.15.46:0/190518569 >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 msgr2=0x7f64a00d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649dd9b640 1 --2- 172.21.15.46:0/190518569 >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 0x7f64a00d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649dd9b640 1 -- 172.21.15.46:0/190518569 >> v1:172.21.15.78:6789/0 conn(0x7f64a00d0f00 legacy=0x7f64a00b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649dd9b640 1 -- 172.21.15.46:0/190518569 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a00d2a40 con 0x7f64a00cfd20 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.055+0000 7f649e59c640 1 --2- 172.21.15.46:0/190518569 >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 0x7f64a00d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f649dd9b640 1 --2- 172.21.15.46:0/190518569 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a00d0160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64880097a0 tx=0x7f648802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95f0518ad1341886 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f649d59a640 1 -- 172.21.15.46:0/190518569 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488035020 con 0x7f64a00cfd20 2024-04-15T15:16:49.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f649d59a640 1 -- 172.21.15.46:0/190518569 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6488031830 con 0x7f64a00cfd20 2024-04-15T15:16:49.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f649d59a640 1 -- 172.21.15.46:0/190518569 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6488031c00 con 0x7f64a00cfd20 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f64a4835640 1 -- 172.21.15.46:0/190518569 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 msgr2=0x7f64a00d0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f64a4835640 1 --2- 172.21.15.46:0/190518569 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a00d0160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64880097a0 tx=0x7f648802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f64a4835640 1 -- 172.21.15.46:0/190518569 shutdown_connections 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f64a4835640 1 --2- 172.21.15.46:0/190518569 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a00d0160 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f64a4835640 1 --2- 172.21.15.46:0/190518569 >> v2:172.21.15.78:3300/0 conn(0x7f64a015e790 0x7f64a00d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.056+0000 7f64a4835640 1 -- 172.21.15.46:0/190518569 >> 172.21.15.46:0/190518569 conn(0x7f64a008bf60 msgr2=0x7f64a00d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.057+0000 7f64a4835640 1 -- 172.21.15.46:0/190518569 shutdown_connections 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.057+0000 7f64a4835640 1 -- 172.21.15.46:0/190518569 wait complete. 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.058+0000 7f64a4835640 1 Processor -- start 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.058+0000 7f64a4835640 1 -- start start 2024-04-15T15:16:49.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a015c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a00d0f00 0x7f64a0152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 0x7f64a00c89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64a015ecf0 con 0x7f64a015e790 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f64a015eb70 con 0x7f64a00d0f00 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f64a00d13b0 con 0x7f64a00cfd20 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 0x7f64a00c89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 0x7f64a00c89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41710/0 (socket says 172.21.15.46:41710) 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 -- 172.21.15.46:0/289710919 learned_addr learned my addr 172.21.15.46:0/289710919 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649dd9b640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a00d0f00 0x7f64a0152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649e59c640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 -- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 msgr2=0x7f64a015c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a015c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 -- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a00d0f00 msgr2=0x7f64a0152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a00d0f00 0x7f64a0152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 -- 172.21.15.46:0/289710919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a01f8030 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f649ed9d640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 0x7f64a00c89b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f649800c780 tx=0x7f649800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498017070 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649800a640 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a01f8310 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649800aa30 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.059+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a00c81b0 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.060+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a00689e0 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.061+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f649801a070 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.061+0000 7f6496ffd640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f648006c8f0 0x7f648006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.061+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f649808b4e0 con 0x7f64a015e790 2024-04-15T15:16:49.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.062+0000 7f649e59c640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f648006c8f0 0x7f648006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.062+0000 7f649e59c640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f648006c8f0 0x7f648006edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6490006ca0 tx=0x7f6490009240 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.066+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6498058eb0 con 0x7f64a015e790 2024-04-15T15:16:49.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.251+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64a00d1ce0 con 0x7f64a015e790 2024-04-15T15:16:49.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.253+0000 7f6496ffd640 1 -- 172.21.15.46:0/289710919 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18857 (secure 0 0 0) 0x7f649805d1b0 con 0x7f64a015e790 2024-04-15T15:16:49.255 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:49.255 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:19.707153+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:36.512834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:36.868541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:40.569845+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.18","mappings":[{"from":3,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f648006c8f0 msgr2=0x7f648006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f648006c8f0 0x7f648006edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6490006ca0 tx=0x7f6490009240 comp rx=0 tx=0).stop 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 msgr2=0x7f64a00c89b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 0x7f64a00c89b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f649800c780 tx=0x7f649800cc40 comp rx=0 tx=0).stop 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 shutdown_connections 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f648006c8f0 0x7f648006edb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64a015e790 0x7f64a00c89b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64a00d0f00 0x7f64a0152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 --2- 172.21.15.46:0/289710919 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64a00cfd20 0x7f64a015c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 >> 172.21.15.46:0/289710919 conn(0x7f64a008bf60 msgr2=0x7f64a00b89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 shutdown_connections 2024-04-15T15:16:49.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.255+0000 7f64a4835640 1 -- 172.21.15.46:0/289710919 wait complete. 2024-04-15T15:16:49.270 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:16:49.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.374+0000 7f1e8870d640 1 Processor -- start 2024-04-15T15:16:49.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.374+0000 7f1e8870d640 1 -- start start 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.375+0000 7f1e8870d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f1e800b49d0 0x7f1e80160af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.375+0000 7f1e8870d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e800d14b0 con 0x7f1e8009ac40 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.375+0000 7f1e8870d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e800cd550 con 0x7f1e8009b020 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.375+0000 7f1e8870d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f1e800b1b70 con 0x7f1e800b49d0 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.375+0000 7f1e864a9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1e8009ac40 0x7f1e800d0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:40482/0 (socket says 172.21.15.46:40482) 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.375+0000 7f1e864a9640 1 -- 172.21.15.46:0/3131052869 learned_addr learned my addr 172.21.15.46:0/3131052869 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e85ca8640 1 --2- 172.21.15.46:0/3131052869 >> v2:172.21.15.78:3300/0 conn(0x7f1e800b49d0 0x7f1e80160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879807022 0 0) 0x7f1e800d14b0 con 0x7f1e8009ac40 2024-04-15T15:16:49.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e6c003520 con 0x7f1e8009ac40 2024-04-15T15:16:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 336195656 0 0) 0x7f1e800cd550 con 0x7f1e8009b020 2024-04-15T15:16:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e800d14b0 con 0x7f1e8009b020 2024-04-15T15:16:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e85ca8640 1 -- 172.21.15.46:0/3131052869 >> v1:172.21.15.78:6789/0 conn(0x7f1e8009b020 legacy=0x7f1e800ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e85ca8640 1 -- 172.21.15.46:0/3131052869 >> v1:172.21.15.46:6789/0 conn(0x7f1e8009ac40 legacy=0x7f1e800d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 470553087 0 0) 0x7f1e6c003520 con 0x7f1e8009ac40 2024-04-15T15:16:49.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.376+0000 7f1e85ca8640 1 -- 172.21.15.46:0/3131052869 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e800cd830 con 0x7f1e800b49d0 2024-04-15T15:16:49.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.377+0000 7f1e85ca8640 1 --2- 172.21.15.46:0/3131052869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e800b49d0 0x7f1e80160af0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1e700097a0 tx=0x7f1e7002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a7e88b2e888802a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.377+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c001670 con 0x7f1e800b49d0 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.377+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e70037070 con 0x7f1e800b49d0 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.377+0000 7f1e854a7640 1 -- 172.21.15.46:0/3131052869 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e70031790 con 0x7f1e800b49d0 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.377+0000 7f1e8870d640 1 -- 172.21.15.46:0/3131052869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e800b49d0 msgr2=0x7f1e80160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.377+0000 7f1e8870d640 1 --2- 172.21.15.46:0/3131052869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e800b49d0 0x7f1e80160af0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1e700097a0 tx=0x7f1e7002b8b0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.378+0000 7f1e8870d640 1 -- 172.21.15.46:0/3131052869 shutdown_connections 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.378+0000 7f1e8870d640 1 --2- 172.21.15.46:0/3131052869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e800b49d0 0x7f1e80160af0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.378+0000 7f1e8870d640 1 -- 172.21.15.46:0/3131052869 >> 172.21.15.46:0/3131052869 conn(0x7f1e800b3eb0 msgr2=0x7f1e800b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.378+0000 7f1e8870d640 1 -- 172.21.15.46:0/3131052869 shutdown_connections 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.378+0000 7f1e8870d640 1 -- 172.21.15.46:0/3131052869 wait complete. 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 Processor -- start 2024-04-15T15:16:49.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 -- start start 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 0x7f1e80158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8009b020 0x7f1e80158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1e800b49d0 0x7f1e800cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1e80161030 con 0x7f1e8009ac40 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f1e800cd1e0 con 0x7f1e8009b020 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.379+0000 7f1e8870d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1e800b1d20 con 0x7f1e800b49d0 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 0x7f1e80158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 0x7f1e80158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41720/0 (socket says 172.21.15.46:41720) 2024-04-15T15:16:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 -- 172.21.15.46:0/1719582522 learned_addr learned my addr 172.21.15.46:0/1719582522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e85ca8640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8009b020 0x7f1e80158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e86caa640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1e800b49d0 0x7f1e800cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 -- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1e800b49d0 msgr2=0x7f1e800cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1e800b49d0 0x7f1e800cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 -- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8009b020 msgr2=0x7f1e80158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8009b020 0x7f1e80158590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 -- 172.21.15.46:0/1719582522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e800c5ef0 con 0x7f1e8009ac40 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e86caa640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1e800b49d0 0x7f1e800cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e85ca8640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8009b020 0x7f1e80158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e864a9640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 0x7f1e80158e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c780 tx=0x7f1e7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c017070 con 0x7f1e8009ac40 2024-04-15T15:16:49.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e7c005ad0 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e800ca0f0 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c007070 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.380+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e800ca6b0 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.381+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e800689e0 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.382+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1e7c01a070 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.382+0000 7f1e76ffd640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1e6406c920 0x7f1e6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.382+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1e7c08b410 con 0x7f1e8009ac40 2024-04-15T15:16:49.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.382+0000 7f1e85ca8640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1e6406c920 0x7f1e6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.383+0000 7f1e85ca8640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1e6406c920 0x7f1e6406ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e70036000 tx=0x7f1e70008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.386+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e800c5ef0 con 0x7f1e8009ac40 2024-04-15T15:16:49.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.613+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1e8015eeb0 con 0x7f1e8009ac40 2024-04-15T15:16:49.615 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-04-15T15:16:49.613+0000 7fb7c20f9640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:16:49.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.614+0000 7f1e76ffd640 1 -- 172.21.15.46:0/1719582522 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f1e7c0594d0 con 0x7f1e8009ac40 2024-04-15T15:16:49.616 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1e6406c920 msgr2=0x7f1e6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1e6406c920 0x7f1e6406ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1e70036000 tx=0x7f1e70008030 comp rx=0 tx=0).stop 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 msgr2=0x7f1e80158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 0x7f1e80158e30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c00c780 tx=0x7f1e7c00cc40 comp rx=0 tx=0).stop 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 shutdown_connections 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f1e6406c920 0x7f1e6406ede0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1e800b49d0 0x7f1e800cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e8009b020 0x7f1e80158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 --2- 172.21.15.46:0/1719582522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e8009ac40 0x7f1e80158e30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 >> 172.21.15.46:0/1719582522 conn(0x7f1e800b3eb0 msgr2=0x7f1e800af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 shutdown_connections 2024-04-15T15:16:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.617+0000 7f1e8870d640 1 -- 172.21.15.46:0/1719582522 wait complete. 2024-04-15T15:16:49.621 INFO:tasks.ceph.mds.b.smithi078.stderr: -6027> 2024-04-15T15:15:31.975+0000 7fabe762f640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-15T15:16:49.622 INFO:tasks.ceph.mds.b.smithi078.stderr: -4898> 2024-04-15T15:15:38.969+0000 7fabe1623640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:16:49.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:16:49.654 INFO:tasks.ceph.mds.b.smithi078.stdout:starting mds.b at 2024-04-15T15:16:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.734+0000 7ff57cf96640 1 Processor -- start 2024-04-15T15:16:49.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.734+0000 7ff57cf96640 1 -- start start 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff57cf96640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 0x7ff5780b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff57cf96640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff5780d3270 0x7ff5780cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff57cf96640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5780b47a0 con 0x7ff5780d2360 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff57cf96640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7ff5780b53b0 con 0x7ff5780d3270 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff57cf96640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff5780b4a80 con 0x7ff57815e700 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff575d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff5780d2360 0x7ff5780d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35424/0 (socket says 172.21.15.46:35424) 2024-04-15T15:16:49.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.735+0000 7ff575d9b640 1 -- 172.21.15.46:0/3132887518 learned_addr learned my addr 172.21.15.46:0/3132887518 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57659c640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 0x7ff5780b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff576d9d640 1 --2- 172.21.15.46:0/3132887518 >> v2:172.21.15.78:3300/0 conn(0x7ff5780d3270 0x7ff5780cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847589438 0 0) 0x7ff5780b47a0 con 0x7ff5780d2360 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff564003520 con 0x7ff5780d2360 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2605419864 0 0) 0x7ff564003520 con 0x7ff5780d2360 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 msgr2=0x7ff5780b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 0x7ff5780b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 >> v2:172.21.15.78:3300/0 conn(0x7ff5780d3270 msgr2=0x7ff5780cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 --2- 172.21.15.46:0/3132887518 >> v2:172.21.15.78:3300/0 conn(0x7ff5780d3270 0x7ff5780cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5780b4d60 con 0x7ff5780d2360 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7ff560001670 con 0x7ff5780d2360 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 >> v1:172.21.15.46:6789/0 conn(0x7ff5780d2360 legacy=0x7ff5780d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 0x7ff5640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff564003f30 0x7ff564004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff564004d40 0x7ff5640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780b53b0 con 0x7ff564003f30 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780b4a80 con 0x7ff564004d40 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff56400b9f0 con 0x7ff57815e700 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff575d9b640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff564003f30 0x7ff564004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff57659c640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 0x7ff5640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff576d9d640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff564004d40 0x7ff5640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff575d9b640 1 -- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 msgr2=0x7ff5640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff575d9b640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff57815e700 0x7ff5640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff575d9b640 1 -- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff564004d40 msgr2=0x7ff5640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff575d9b640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff564004d40 0x7ff5640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.736+0000 7ff575d9b640 1 -- 172.21.15.46:0/3132887518 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff564003520 con 0x7ff564003f30 2024-04-15T15:16:49.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.737+0000 7ff575d9b640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff564003f30 0x7ff564004800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c240 tx=0x7ff564005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd10ca203e4bde05 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.737+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560007d30 con 0x7ff564003f30 2024-04-15T15:16:49.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.737+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff560030070 con 0x7ff564003f30 2024-04-15T15:16:49.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.737+0000 7ff57559a640 1 -- 172.21.15.46:0/3132887518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56002cda0 con 0x7ff564003f30 2024-04-15T15:16:49.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff57cf96640 1 -- 172.21.15.46:0/3132887518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff564003f30 msgr2=0x7ff564004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3132887518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff564003f30 0x7ff564004800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff56000c240 tx=0x7ff564005830 comp rx=0 tx=0).stop 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff576d9d640 1 -- 172.21.15.46:0/3132887518 reap_dead start 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff57cf96640 1 -- 172.21.15.46:0/3132887518 shutdown_connections 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff57cf96640 1 -- 172.21.15.46:0/3132887518 >> 172.21.15.46:0/3132887518 conn(0x7ff57808bf60 msgr2=0x7ff5780d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff57cf96640 1 -- 172.21.15.46:0/3132887518 shutdown_connections 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.738+0000 7ff57cf96640 1 -- 172.21.15.46:0/3132887518 wait complete. 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.739+0000 7ff57cf96640 1 Processor -- start 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.739+0000 7ff57cf96640 1 -- start start 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57cf96640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff564003f30 0x7ff5780cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57cf96640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff57814e720 0x7ff5780cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57cf96640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 0x7ff5780cf850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57cf96640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780d1d40 con 0x7ff5780cc230 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57cf96640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff5780b4450 con 0x7ff57814e720 2024-04-15T15:16:49.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57cf96640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff5780d1ec0 con 0x7ff564003f30 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 0x7ff5780cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 0x7ff5780cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47522/0 (socket says 172.21.15.46:47522) 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 -- 172.21.15.46:0/3995851709 learned_addr learned my addr 172.21.15.46:0/3995851709 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff575d9b640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff57814e720 0x7ff5780cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff57659c640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff564003f30 0x7ff5780cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 -- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff564003f30 msgr2=0x7ff5780cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff564003f30 0x7ff5780cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 -- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff57814e720 msgr2=0x7ff5780cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff57814e720 0x7ff5780cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.740+0000 7ff576d9d640 1 -- 172.21.15.46:0/3995851709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5780cc6b0 con 0x7ff5780cc230 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.741+0000 7ff576d9d640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 0x7ff5780cf850 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff568011860 tx=0x7ff568011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.741+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568019020 con 0x7ff5780cc230 2024-04-15T15:16:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.741+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff568011e50 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.741+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5780cc950 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.741+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff568004460 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.741+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5780ccf10 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.742+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5780689e0 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.742+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff56801c020 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.743+0000 7ff56effd640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff55806c8f0 0x7ff55806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.743+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7ff56808c150 con 0x7ff5780cc230 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.743+0000 7ff57659c640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff55806c8f0 0x7ff55806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.743+0000 7ff57659c640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff55806c8f0 0x7ff55806edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff570014510 tx=0x7ff57000b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.747+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff568023020 con 0x7ff5780cc230 2024-04-15T15:16:49.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:49.971+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff5780ba500 con 0x7ff5780cc230 2024-04-15T15:16:50.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.616+0000 7ff56effd640 1 -- 172.21.15.46:0/3995851709 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7ff568059b60 con 0x7ff5780cc230 2024-04-15T15:16:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff55806c8f0 msgr2=0x7ff55806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff55806c8f0 0x7ff55806edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff570014510 tx=0x7ff57000b040 comp rx=0 tx=0).stop 2024-04-15T15:16:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 msgr2=0x7ff5780cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 0x7ff5780cf850 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff568011860 tx=0x7ff568011d20 comp rx=0 tx=0).stop 2024-04-15T15:16:50.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 shutdown_connections 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff55806c8f0 0x7ff55806edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5780cc230 0x7ff5780cf850 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff57814e720 0x7ff5780cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 --2- 172.21.15.46:0/3995851709 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff564003f30 0x7ff5780cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 >> 172.21.15.46:0/3995851709 conn(0x7ff57808bf60 msgr2=0x7ff5780c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 shutdown_connections 2024-04-15T15:16:50.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.618+0000 7ff57cf96640 1 -- 172.21.15.46:0/3995851709 wait complete. 2024-04-15T15:16:50.632 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:16:50.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.739+0000 7f25b06f2640 1 Processor -- start 2024-04-15T15:16:50.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.739+0000 7f25b06f2640 1 -- start start 2024-04-15T15:16:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25b06f2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25a815acb0 0x7f25a80ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25b06f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a80cf960 0x7f25a80cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25b06f2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25a80c60e0 con 0x7f25a80b4b50 2024-04-15T15:16:50.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25b06f2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f25a80ce460 con 0x7f25a815acb0 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25b06f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f25a80cdea0 con 0x7f25a80cf960 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25a815acb0 0x7f25a80ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25a815acb0 0x7f25a80ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47532/0 (socket says 172.21.15.46:47532) 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 -- 172.21.15.46:0/3244813787 learned_addr learned my addr 172.21.15.46:0/3244813787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 -- 172.21.15.46:0/3244813787 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a80cf960 msgr2=0x7f25a80cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 --2- 172.21.15.46:0/3244813787 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a80cf960 0x7f25a80cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 -- 172.21.15.46:0/3244813787 >> v1:172.21.15.78:6789/0 conn(0x7f25a80b4b50 legacy=0x7f25a815e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.740+0000 7f25ae48e640 1 -- 172.21.15.46:0/3244813787 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a80ce180 con 0x7f25a815acb0 2024-04-15T15:16:50.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.741+0000 7f25ae48e640 1 --2- 172.21.15.46:0/3244813787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a815acb0 0x7f25a80ccb80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009870 tx=0x7f25a402ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f82d224cd8cf6f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.741+0000 7f25ad48c640 1 -- 172.21.15.46:0/3244813787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a4030820 con 0x7f25a815acb0 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.741+0000 7f25ad48c640 1 -- 172.21.15.46:0/3244813787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a40309c0 con 0x7f25a815acb0 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.741+0000 7f25ad48c640 1 -- 172.21.15.46:0/3244813787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a4030d90 con 0x7f25a815acb0 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 -- 172.21.15.46:0/3244813787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a815acb0 msgr2=0x7f25a80ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 --2- 172.21.15.46:0/3244813787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a815acb0 0x7f25a80ccb80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009870 tx=0x7f25a402ca00 comp rx=0 tx=0).stop 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 -- 172.21.15.46:0/3244813787 shutdown_connections 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 --2- 172.21.15.46:0/3244813787 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a80cf960 0x7f25a80cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 --2- 172.21.15.46:0/3244813787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a815acb0 0x7f25a80ccb80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 -- 172.21.15.46:0/3244813787 >> 172.21.15.46:0/3244813787 conn(0x7f25a806bbc0 msgr2=0x7f25a806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 -- 172.21.15.46:0/3244813787 shutdown_connections 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.742+0000 7f25b06f2640 1 -- 172.21.15.46:0/3244813787 wait complete. 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.743+0000 7f25b06f2640 1 Processor -- start 2024-04-15T15:16:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.743+0000 7f25b06f2640 1 -- start start 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25b06f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 0x7f25a814fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25b06f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80cf960 0x7f25a80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25b06f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a815acb0 0x7f25a80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25b06f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25a80cfed0 con 0x7f25a80b4b50 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25b06f2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f25a80b4130 con 0x7f25a80cf960 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25b06f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f25a815b090 con 0x7f25a815acb0 2024-04-15T15:16:50.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 0x7f25a814fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 0x7f25a814fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47534/0 (socket says 172.21.15.46:47534) 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 -- 172.21.15.46:0/2027607558 learned_addr learned my addr 172.21.15.46:0/2027607558 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25adc8d640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80cf960 0x7f25a80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25aec8f640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a815acb0 0x7f25a80b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 -- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a815acb0 msgr2=0x7f25a80b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a815acb0 0x7f25a80b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 -- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80cf960 msgr2=0x7f25a80b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80cf960 0x7f25a80b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 -- 172.21.15.46:0/2027607558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a81500c0 con 0x7f25a80b4b50 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25adc8d640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80cf960 0x7f25a80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25aec8f640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a815acb0 0x7f25a80b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f25ae48e640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 0x7f25a814fb80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009870 tx=0x7f25a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.744+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a4047070 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.745+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a4038bd0 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.745+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a815cb60 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.745+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a40423f0 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.745+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a815d120 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.745+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a80689e0 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.746+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f25a404a070 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.746+0000 7f259effd640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f258c06c8f0 0x7f258c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.746+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f25a40add00 con 0x7f25a80b4b50 2024-04-15T15:16:50.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.747+0000 7f25adc8d640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f258c06c8f0 0x7f258c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.747+0000 7f25adc8d640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f258c06c8f0 0x7f258c06edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2598007fd0 tx=0x7f2598009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.751+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f25a4034030 con 0x7f25a80b4b50 2024-04-15T15:16:50.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:50.938+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f25a80b0d10 con 0x7f25a80b4b50 2024-04-15T15:16:51.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.818+0000 7f259effd640 1 -- 172.21.15.46:0/2027607558 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7f25a407bcd0 con 0x7f25a80b4b50 2024-04-15T15:16:51.820 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:16:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f258c06c8f0 msgr2=0x7f258c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f258c06c8f0 0x7f258c06edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2598007fd0 tx=0x7f2598009040 comp rx=0 tx=0).stop 2024-04-15T15:16:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 msgr2=0x7f25a814fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 0x7f25a814fb80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009870 tx=0x7f25a4004730 comp rx=0 tx=0).stop 2024-04-15T15:16:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 shutdown_connections 2024-04-15T15:16:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f258c06c8f0 0x7f258c06edb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25a815acb0 0x7f25a80b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25a80cf960 0x7f25a80b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 --2- 172.21.15.46:0/2027607558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25a80b4b50 0x7f25a814fb80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.822+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 >> 172.21.15.46:0/2027607558 conn(0x7f25a806bbc0 msgr2=0x7f25a80af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.823+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 shutdown_connections 2024-04-15T15:16:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.823+0000 7f25b06f2640 1 -- 172.21.15.46:0/2027607558 wait complete. 2024-04-15T15:16:51.839 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:16:51.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.953+0000 7f4a6a606640 1 Processor -- start 2024-04-15T15:16:51.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.953+0000 7f4a6a606640 1 -- start start 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.953+0000 7f4a6a606640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4a640b35a0 0x7f4a6406d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.953+0000 7f4a6a606640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a6409b480 con 0x7f4a6406d950 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.953+0000 7f4a6a606640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a6409b760 con 0x7f4a6415e860 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.953+0000 7f4a6a606640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f4a6409bd20 con 0x7f4a640b35a0 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a68ba3640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4a6406d950 0x7f4a640ccdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35432/0 (socket says 172.21.15.46:35432) 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a68ba3640 1 -- 172.21.15.46:0/3325197587 learned_addr learned my addr 172.21.15.46:0/3325197587 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:51.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/3325197587 >> v2:172.21.15.78:3300/0 conn(0x7f4a640b35a0 0x7f4a6406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069883669 0 0) 0x7f4a6409b480 con 0x7f4a6406d950 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a50003520 con 0x7f4a6406d950 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950802421 0 0) 0x7f4a6409b760 con 0x7f4a6415e860 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a6409b480 con 0x7f4a6415e860 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2232857874 0 0) 0x7f4a50003520 con 0x7f4a6406d950 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 >> v2:172.21.15.78:3300/0 conn(0x7f4a640b35a0 msgr2=0x7f4a6406d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 --2- 172.21.15.46:0/3325197587 >> v2:172.21.15.78:3300/0 conn(0x7f4a640b35a0 0x7f4a6406d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 >> v1:172.21.15.78:6789/0 conn(0x7f4a6415e860 legacy=0x7f4a6406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6409ba40 con 0x7f4a6406d950 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f4a5c0019f0 con 0x7f4a6406d950 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 >> v1:172.21.15.46:6789/0 conn(0x7f4a6406d950 legacy=0x7f4a640ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a50003b90 0x7f4a50003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a500051c0 0x7f4a5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6409bd20 con 0x7f4a50004800 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5000c0c0 con 0x7f4a50003b90 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5000c2c0 con 0x7f4a500051c0 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a63fff640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a50003b90 0x7f4a50003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.954+0000 7f4a68ba3640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a500051c0 0x7f4a5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a5bfff640 1 -- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a500051c0 msgr2=0x7f4a5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a500051c0 0x7f4a5000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a5bfff640 1 -- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a50003b90 msgr2=0x7f4a50003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a50003b90 0x7f4a50003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a5bfff640 1 -- 172.21.15.46:0/3325197587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a50003520 con 0x7f4a50004800 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a68ba3640 1 -- 172.21.15.46:0/3325197587 reap_dead start 2024-04-15T15:16:51.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.955+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009ac0 tx=0x7f4a50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf067dd38a7e108 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c039070 con 0x7f4a50004800 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a4c02c6b0 con 0x7f4a50004800 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a637fe640 1 -- 172.21.15.46:0/3325197587 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c009690 con 0x7f4a50004800 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 -- 172.21.15.46:0/3325197587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 msgr2=0x7f4a50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009ac0 tx=0x7f4a50005830 comp rx=0 tx=0).stop 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 -- 172.21.15.46:0/3325197587 shutdown_connections 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 --2- 172.21.15.46:0/3325197587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a50004c80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 -- 172.21.15.46:0/3325197587 >> 172.21.15.46:0/3325197587 conn(0x7f4a640b3f20 msgr2=0x7f4a640b4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 -- 172.21.15.46:0/3325197587 shutdown_connections 2024-04-15T15:16:51.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.956+0000 7f4a6a606640 1 -- 172.21.15.46:0/3325197587 wait complete. 2024-04-15T15:16:51.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.957+0000 7f4a6a606640 1 Processor -- start 2024-04-15T15:16:51.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.957+0000 7f4a6a606640 1 -- start start 2024-04-15T15:16:51.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a6a606640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a641f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a6a606640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640b59f0 0x7f4a641f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a6a606640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d0cc0 0x7f4a641f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a6a606640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6415ec40 con 0x7f4a50004800 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a6a606640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4a6406de10 con 0x7f4a640d0cc0 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a6a606640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4a6415edc0 con 0x7f4a640b59f0 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a641f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a63fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a641f43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47546/0 (socket says 172.21.15.46:47546) 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a63fff640 1 -- 172.21.15.46:0/2238621684 learned_addr learned my addr 172.21.15.46:0/2238621684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640b59f0 0x7f4a641f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.958+0000 7f4a68ba3640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d0cc0 0x7f4a641f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a63fff640 1 -- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640b59f0 msgr2=0x7f4a641f5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a63fff640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640b59f0 0x7f4a641f5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a63fff640 1 -- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d0cc0 msgr2=0x7f4a641f1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a63fff640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d0cc0 0x7f4a641f1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a63fff640 1 -- 172.21.15.46:0/2238621684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a640c7220 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a63fff640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a641f43a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400f7a0 tx=0x7f4a5400fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a5401a070 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a54005110 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a641f49f0 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a54007050 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.959+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a641f4fb0 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.960+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a6406cd00 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.960+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a5401d070 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.961+0000 7f4a617fa640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.961+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6383+0+0 (secure 0 0 0) 0x7f4a5408b560 con 0x7f4a50004800 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.961+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.962+0000 7f4a5bfff640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c036000 tx=0x7f4a4c030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:51.965+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a6406cd00 con 0x7f4a50004800 2024-04-15T15:16:52.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.149+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4a64069480 con 0x7f4a50004800 2024-04-15T15:16:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.824+0000 7f4a617fa640 1 -- 172.21.15.46:0/2238621684 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7f4a64069480 con 0x7f4a50004800 2024-04-15T15:16:52.826 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.826+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 msgr2=0x7f4a4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.826+0000 7f4a6a606640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c036000 tx=0x7f4a4c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.826+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 msgr2=0x7f4a641f43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.826+0000 7f4a6a606640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a641f43a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400f7a0 tx=0x7f4a5400fba0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 shutdown_connections 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4a4406c8f0 0x7f4a4406edb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4a640d0cc0 0x7f4a641f1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4a640b59f0 0x7f4a641f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 --2- 172.21.15.46:0/2238621684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a50004800 0x7f4a641f43a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 >> 172.21.15.46:0/2238621684 conn(0x7f4a640b3f20 msgr2=0x7f4a6415dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 shutdown_connections 2024-04-15T15:16:52.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.827+0000 7f4a6a606640 1 -- 172.21.15.46:0/2238621684 wait complete. 2024-04-15T15:16:52.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-15T15:16:52.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.956+0000 7fe8a01a8640 1 Processor -- start 2024-04-15T15:16:52.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.956+0000 7fe8a01a8640 1 -- start start 2024-04-15T15:16:52.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe8a01a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe8980ba260 0x7fe8980d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe8a01a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe89815ed10 con 0x7fe8980d16e0 2024-04-15T15:16:52.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe8a01a8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980b7fb0 con 0x7fe89815e930 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe8a01a8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe8980b8bc0 con 0x7fe8980ba260 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe89e745640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe8980d16e0 0x7fe8980b9050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35448/0 (socket says 172.21.15.46:35448) 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe89e745640 1 -- 172.21.15.46:0/61795204 learned_addr learned my addr 172.21.15.46:0/61795204 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe89d743640 1 --2- 172.21.15.46:0/61795204 >> v2:172.21.15.78:3300/0 conn(0x7fe8980ba260 0x7fe8980d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe89cf42640 1 -- 172.21.15.46:0/61795204 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571285419 0 0) 0x7fe89815ed10 con 0x7fe8980d16e0 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.957+0000 7fe89cf42640 1 -- 172.21.15.46:0/61795204 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe884003520 con 0x7fe8980d16e0 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.958+0000 7fe89d743640 1 -- 172.21.15.46:0/61795204 >> v1:172.21.15.78:6789/0 conn(0x7fe89815e930 legacy=0x7fe8980b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.958+0000 7fe89d743640 1 -- 172.21.15.46:0/61795204 >> v1:172.21.15.46:6789/0 conn(0x7fe8980d16e0 legacy=0x7fe8980b9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.958+0000 7fe89d743640 1 -- 172.21.15.46:0/61795204 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8980b8290 con 0x7fe8980ba260 2024-04-15T15:16:52.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.958+0000 7fe89d743640 1 --2- 172.21.15.46:0/61795204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8980ba260 0x7fe8980d2a60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe8980b8d40 tx=0x7fe88802b9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebe888f32f4fb35c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe89cf42640 1 -- 172.21.15.46:0/61795204 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888034070 con 0x7fe8980ba260 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe89cf42640 1 -- 172.21.15.46:0/61795204 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe888030890 con 0x7fe8980ba260 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe89cf42640 1 -- 172.21.15.46:0/61795204 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888030c40 con 0x7fe8980ba260 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 -- 172.21.15.46:0/61795204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8980ba260 msgr2=0x7fe8980d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/61795204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8980ba260 0x7fe8980d2a60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe8980b8d40 tx=0x7fe88802b9d0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 -- 172.21.15.46:0/61795204 shutdown_connections 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/61795204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8980ba260 0x7fe8980d2a60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 -- 172.21.15.46:0/61795204 >> 172.21.15.46:0/61795204 conn(0x7fe898067b10 msgr2=0x7fe898067f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 -- 172.21.15.46:0/61795204 shutdown_connections 2024-04-15T15:16:52.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.959+0000 7fe8a01a8640 1 -- 172.21.15.46:0/61795204 wait complete. 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.960+0000 7fe8a01a8640 1 Processor -- start 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.960+0000 7fe8a01a8640 1 -- start start 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe8a01a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 0x7fe8980cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe8a01a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8980d16e0 0x7fe898160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe8a01a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89815e930 0x7fe8980ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe8a01a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe8980ba6a0 con 0x7fe8980ba260 2024-04-15T15:16:52.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe8a01a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe8980b8850 con 0x7fe89815e930 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe8a01a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe8980d2fa0 con 0x7fe8980d16e0 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe89df44640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 0x7fe8980cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe89df44640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 0x7fe8980cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47550/0 (socket says 172.21.15.46:47550) 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe89df44640 1 -- 172.21.15.46:0/966482097 learned_addr learned my addr 172.21.15.46:0/966482097 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe89e745640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89815e930 0x7fe8980ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.961+0000 7fe89d743640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8980d16e0 0x7fe898160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89df44640 1 -- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8980d16e0 msgr2=0x7fe898160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89df44640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8980d16e0 0x7fe898160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89df44640 1 -- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89815e930 msgr2=0x7fe8980ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89df44640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89815e930 0x7fe8980ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89df44640 1 -- 172.21.15.46:0/966482097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8981616e0 con 0x7fe8980ba260 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89e745640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89815e930 0x7fe8980ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe89df44640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 0x7fe8980cc4d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c400 tx=0x7fe89400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8940052d0 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe894005490 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8981619c0 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe894007260 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.962+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89815ef10 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.963+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8980b5f90 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.963+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe894007400 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.964+0000 7fe88e7fc640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe87c06c8f0 0x7fe87c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.964+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7fe89408b140 con 0x7fe8980ba260 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.964+0000 7fe89d743640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe87c06c8f0 0x7fe87c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.965+0000 7fe89d743640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe87c06c8f0 0x7fe87c06edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe888036000 tx=0x7fe888008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:52.968+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8980b5f90 con 0x7fe8980ba260 2024-04-15T15:16:53.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.160+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe8980d2a60 con 0x7fe8980ba260 2024-04-15T15:16:53.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.825+0000 7fe88e7fc640 1 -- 172.21.15.46:0/966482097 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7fe8980d2a60 con 0x7fe8980ba260 2024-04-15T15:16:53.827 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' removed 2024-04-15T15:16:53.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe87c06c8f0 msgr2=0x7fe87c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe87c06c8f0 0x7fe87c06edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe888036000 tx=0x7fe888008030 comp rx=0 tx=0).stop 2024-04-15T15:16:53.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 msgr2=0x7fe8980cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 0x7fe8980cc4d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c400 tx=0x7fe89400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 shutdown_connections 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe87c06c8f0 0x7fe87c06edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe89815e930 0x7fe8980ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe8980d16e0 0x7fe898160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 --2- 172.21.15.46:0/966482097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8980ba260 0x7fe8980cc4d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.829+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 >> 172.21.15.46:0/966482097 conn(0x7fe898067b10 msgr2=0x7fe8980ca280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.830+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 shutdown_connections 2024-04-15T15:16:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:53.830+0000 7fe8a01a8640 1 -- 172.21.15.46:0/966482097 wait complete. 2024-04-15T15:16:53.847 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:16:53.847 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-15T15:16:53.847 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:16:53.847 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:53.848 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:16:53.848 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-15T15:16:53.848 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:16:53.848 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:53.848 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:16:53.848 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-15T15:16:53.848 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:16:53.849 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:53.849 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:16:53.849 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-15T15:16:53.849 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:16:53.849 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:53.864 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:16:53.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:16:53.865 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:16:53.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:16:53.867 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:16:53.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:16:53.868 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:16:53.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:16:53.908 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:16:53.908 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:53.909 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:16:53.909 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:53.910 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:16:53.911 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:53.911 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:16:53.911 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:53.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-15T15:16:53.990 INFO:tasks.ceph.mds.d.smithi078.stdout:starting mds.d at 2024-04-15T15:16:53.991 INFO:tasks.ceph.mds.b.smithi078.stdout:starting mds.b at 2024-04-15T15:16:53.993 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2024-04-15T15:16:53.994 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-04-15T15:16:54.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.022+0000 7f7cce288640 1 Processor -- start 2024-04-15T15:16:54.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.023+0000 7f7cce288640 1 -- start start 2024-04-15T15:16:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.023+0000 7f7cce288640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc815e9a0 0x7f7cc80d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.023+0000 7f7cce288640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7cc80d1400 0x7f7cc80cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.023+0000 7f7cce288640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc80d1e60 con 0x7f7cc80cf750 2024-04-15T15:16:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.023+0000 7f7cce288640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f7cc80d18a0 con 0x7f7cc80d1400 2024-04-15T15:16:54.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.023+0000 7f7cce288640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc80d1b80 con 0x7f7cc815e9a0 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cc7fff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc815e9a0 0x7f7cc80d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cbffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7cc80d1400 0x7f7cc80cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7ccc825640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7cc80cf750 0x7f7cc80d2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:39830/0 (socket says 172.21.15.46:39830) 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7ccc825640 1 -- 172.21.15.46:0/4155968957 learned_addr learned my addr 172.21.15.46:0/4155968957 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cbffff640 1 -- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc815e9a0 msgr2=0x7f7cc80d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cbffff640 1 --2- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc815e9a0 0x7f7cc80d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cbffff640 1 -- 172.21.15.46:0/4155968957 >> v1:172.21.15.78:6789/0 conn(0x7f7cc80cf750 legacy=0x7f7cc80d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cc77fe640 1 -- 172.21.15.46:0/4155968957 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2867642260 0 0) 0x7f7cc80d1e60 con 0x7f7cc80cf750 2024-04-15T15:16:54.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.024+0000 7f7cbffff640 1 -- 172.21.15.46:0/4155968957 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc806cff0 con 0x7f7cc80d1400 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.025+0000 7f7cbffff640 1 --2- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80cf210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00097a0 tx=0x7f7cb002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb9d476a570585a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.025+0000 7f7cc77fe640 1 -- 172.21.15.46:0/4155968957 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0035020 con 0x7f7cc80d1400 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.025+0000 7f7cc77fe640 1 -- 172.21.15.46:0/4155968957 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb0031830 con 0x7f7cc80d1400 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.025+0000 7f7cc77fe640 1 -- 172.21.15.46:0/4155968957 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0031c00 con 0x7f7cc80d1400 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 -- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 msgr2=0x7f7cc80cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 --2- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80cf210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00097a0 tx=0x7f7cb002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 -- 172.21.15.46:0/4155968957 shutdown_connections 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 --2- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80cf210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 --2- 172.21.15.46:0/4155968957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc815e9a0 0x7f7cc80d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 -- 172.21.15.46:0/4155968957 >> 172.21.15.46:0/4155968957 conn(0x7f7cc809d950 msgr2=0x7f7cc808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 -- 172.21.15.46:0/4155968957 shutdown_connections 2024-04-15T15:16:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.026+0000 7f7cce288640 1 -- 172.21.15.46:0/4155968957 wait complete. 2024-04-15T15:16:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.027+0000 7f7cce288640 1 Processor -- start 2024-04-15T15:16:54.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.027+0000 7f7cce288640 1 -- start start 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cce288640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc80cf750 0x7f7cc81f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cce288640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cce288640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 0x7f7cc80cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cce288640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc80d24e0 con 0x7f7cc815e9a0 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cce288640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc80d2360 con 0x7f7cc80d1400 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cce288640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc80d32b0 con 0x7f7cc80cf750 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 0x7f7cc80cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 0x7f7cc80cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47568/0 (socket says 172.21.15.46:47568) 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 -- 172.21.15.46:0/3263585081 learned_addr learned my addr 172.21.15.46:0/3263585081 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cc7fff640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc80cf750 0x7f7cc81f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cbffff640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 -- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc80cf750 msgr2=0x7f7cc81f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc80cf750 0x7f7cc81f43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 -- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 msgr2=0x7f7cc80b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7ccc825640 1 -- 172.21.15.46:0/3263585081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc81f5440 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.028+0000 7f7cc7fff640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc80cf750 0x7f7cc81f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.029+0000 7f7ccc825640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 0x7f7cc80cda10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000c780 tx=0x7f7cc000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.029+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0018070 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.029+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc000a640 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.029+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000aab0 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.029+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc81f48f0 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.029+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc81f4e80 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.030+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc80690f0 con 0x7f7cc815e9a0 2024-04-15T15:16:54.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.030+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7cc000ac50 con 0x7f7cc815e9a0 2024-04-15T15:16:54.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.031+0000 7f7cc57fa640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7ca806c8f0 0x7f7ca806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.031+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f7cc008b030 con 0x7f7cc815e9a0 2024-04-15T15:16:54.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.031+0000 7f7cc7fff640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7ca806c8f0 0x7f7ca806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.031+0000 7f7cc7fff640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7ca806c8f0 0x7f7ca806edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7cc815d5b0 tx=0x7f7cb8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.035+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7cc0058ee0 con 0x7f7cc815e9a0 2024-04-15T15:16:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.232+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7cc8075240 con 0x7f7cc815e9a0 2024-04-15T15:16:54.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.839+0000 7f7cc57fa640 1 -- 172.21.15.46:0/3263585081 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f7cc005d1e0 con 0x7f7cc815e9a0 2024-04-15T15:16:54.841 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.841+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7ca806c8f0 msgr2=0x7f7ca806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.841+0000 7f7cce288640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7ca806c8f0 0x7f7ca806edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7cc815d5b0 tx=0x7f7cb8008040 comp rx=0 tx=0).stop 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.841+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 msgr2=0x7f7cc80cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.841+0000 7f7cce288640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 0x7f7cc80cda10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000c780 tx=0x7f7cc000cc40 comp rx=0 tx=0).stop 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 shutdown_connections 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7ca806c8f0 0x7f7ca806edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cc815e9a0 0x7f7cc80cda10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cc80d1400 0x7f7cc80b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 --2- 172.21.15.46:0/3263585081 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7cc80cf750 0x7f7cc81f43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 >> 172.21.15.46:0/3263585081 conn(0x7f7cc809d950 msgr2=0x7f7cc80b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 shutdown_connections 2024-04-15T15:16:54.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.842+0000 7f7cce288640 1 -- 172.21.15.46:0/3263585081 wait complete. 2024-04-15T15:16:54.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-15T15:16:54.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.967+0000 7f141719f640 1 Processor -- start 2024-04-15T15:16:54.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.967+0000 7f141719f640 1 -- start start 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141719f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 0x7f14100d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141719f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14100d32d0 con 0x7f1410162d70 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141719f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f141015f870 con 0x7f14100d2e40 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141719f640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f1410162290 con 0x7f141015e530 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141599c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1410162d70 0x7f141015e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35468/0 (socket says 172.21.15.46:35468) 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141599c640 1 -- 172.21.15.46:0/3000455589 learned_addr learned my addr 172.21.15.46:0/3000455589 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141619d640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 0x7f14100d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1078028988 0 0) 0x7f14100d32d0 con 0x7f1410162d70 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13fc003520 con 0x7f1410162d70 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2156903363 0 0) 0x7f13fc003520 con 0x7f1410162d70 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 msgr2=0x7f14100d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 0x7f14100d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 >> v1:172.21.15.78:6789/0 conn(0x7f14100d2e40 legacy=0x7f141015f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1410162570 con 0x7f1410162d70 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f14000017f0 con 0x7f1410162d70 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 >> v1:172.21.15.46:6789/0 conn(0x7f1410162d70 legacy=0x7f141015e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.968+0000 7f141519b640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0039e0 0x7f13fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141519b640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 0x7f13fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141519b640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13fc004d40 0x7f13fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1410162290 con 0x7f13fc0039e0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc00b9c0 con 0x7f13fc004d40 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f13fc00bbc0 con 0x7f141015e530 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141619d640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0039e0 0x7f13fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141599c640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 0x7f13fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141699e640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13fc004d40 0x7f13fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141619d640 1 -- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 msgr2=0x7f13fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141619d640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f141015e530 0x7f13fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141619d640 1 -- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13fc004d40 msgr2=0x7f13fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141619d640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13fc004d40 0x7f13fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141619d640 1 -- 172.21.15.46:0/3000455589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc003520 con 0x7f13fc0039e0 2024-04-15T15:16:54.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.969+0000 7f141699e640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13fc004d40 0x7f13fc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141619d640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0039e0 0x7f13fc003dc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ac0 tx=0x7f13fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1b0389e8a17101 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c039070 con 0x7f13fc0039e0 2024-04-15T15:16:54.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140c02c600 con 0x7f13fc0039e0 2024-04-15T15:16:54.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141519b640 1 -- 172.21.15.46:0/3000455589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c009690 con 0x7f13fc0039e0 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141719f640 1 -- 172.21.15.46:0/3000455589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0039e0 msgr2=0x7f13fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141719f640 1 --2- 172.21.15.46:0/3000455589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13fc0039e0 0x7f13fc003dc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f140c009ac0 tx=0x7f13fc005830 comp rx=0 tx=0).stop 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141699e640 1 -- 172.21.15.46:0/3000455589 reap_dead start 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141719f640 1 -- 172.21.15.46:0/3000455589 shutdown_connections 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141719f640 1 -- 172.21.15.46:0/3000455589 >> 172.21.15.46:0/3000455589 conn(0x7f141009d780 msgr2=0x7f141008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141719f640 1 -- 172.21.15.46:0/3000455589 shutdown_connections 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.970+0000 7f141719f640 1 -- 172.21.15.46:0/3000455589 wait complete. 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 Processor -- start 2024-04-15T15:16:54.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 -- start start 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13fc0039e0 0x7f14100cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 0x7f14100cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14100cd6a0 0x7f14100cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14100d2a80 con 0x7f14100ca7e0 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f14100d2900 con 0x7f14100cd6a0 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141719f640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f14100d2c00 con 0x7f13fc0039e0 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.972+0000 7f141599c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 0x7f14100cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 0x7f14100cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47598/0 (socket says 172.21.15.46:47598) 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 -- 172.21.15.46:0/2176239784 learned_addr learned my addr 172.21.15.46:0/2176239784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141619d640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13fc0039e0 0x7f14100cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141699e640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14100cd6a0 0x7f14100cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 -- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13fc0039e0 msgr2=0x7f14100cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13fc0039e0 0x7f14100cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 -- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14100cd6a0 msgr2=0x7f14100cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14100cd6a0 0x7f14100cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 -- 172.21.15.46:0/2176239784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14100ce060 con 0x7f14100ca7e0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141699e640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14100cd6a0 0x7f14100cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141599c640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 0x7f14100cd160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f140000f8e0 tx=0x7f140000fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14000072e0 con 0x7f14100ca7e0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1400018070 con 0x7f14100ca7e0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14100b4e70 con 0x7f14100ca7e0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14000057a0 con 0x7f14100ca7e0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.973+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14100b5430 con 0x7f14100ca7e0 2024-04-15T15:16:54.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.974+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141004fd50 con 0x7f14100ca7e0 2024-04-15T15:16:54.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.975+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1400005940 con 0x7f14100ca7e0 2024-04-15T15:16:54.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.975+0000 7f1406ffd640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13f406c920 0x7f13f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.975+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f140008baf0 con 0x7f14100ca7e0 2024-04-15T15:16:54.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.975+0000 7f141619d640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13f406c920 0x7f13f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.976+0000 7f141619d640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13f406c920 0x7f13f406ede0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f140c036000 tx=0x7f140c030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:54.980+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f140005a050 con 0x7f14100ca7e0 2024-04-15T15:16:55.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.238+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f14100b3000 con 0x7f14100ca7e0 2024-04-15T15:16:55.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.238+0000 7f1406ffd640 1 -- 172.21.15.46:0/2176239784 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f1400059950 con 0x7f14100ca7e0 2024-04-15T15:16:55.240 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:55.240 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1mf25mBxAARBEukB5oxK15VCtiDQswRA==", 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2024-04-15T15:16:55.241 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1matdFFBAAygX2/zcEaHpiYvVmgO7aAg==", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1mw6IJDxAAnMgkvYN8tpc8lzkKtvdHfQ==", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-04-15T15:16:55.242 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1m8wrkGxAAQsEA5qc7LD6mnJwLrlZ79Q==", 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBkRB1m+syUBhAA7JRJ68ztnjHCymDvmiEn/A==", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBnRB1mb0s/BRAA0klngWpZ4czyMeWil60J9Q==", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.244 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBqRB1mRCF1AxAA5jX9o6usFucYJoqv0lCYOg==", 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.245 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBtRB1mb/P8ARAAyWXAjjMCwEdGc31rhPSsQg==", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBzRB1mZQA1CxAAlLXltfdCGtIXiPjxx9QNWA==", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.246 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQB2RB1mzBcsChAALZDAAAE8bfcxd6ZCqrbTZQ==", 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.247 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQB5RB1mHYBlCBAAQ0q/qP2uBkHQ64gpR1n5vg==", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQB8RB1mPvq3BhAALSJbv9Hdh+k0E/ZM1f9QSQ==", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.248 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1mChVZIBAAvN0/KdZG4Oam2AdTqRjJtw==", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1mkD/bKRAAmlMhj1H+/gj3v/l9r8wKkQ==", 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1m9TZLJhAAV1mJUfjsEt4HMgYroyY/cg==", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-04-15T15:16:55.250 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1mDFJqLRAAkSUK13PRVTZMg2rT0wIXEg==", 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBhRB1mhVF4JRAAinjmc1A8gU+nsIZDInTkKQ==", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mds", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQCCRB1m31CvBRAABF5EdIt7RAh1V/wer7F9Pw==", 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mds" 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.252 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mgr", 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQCCRB1mZVivBRAAEjLz86chwcF00bLNuCGYVg==", 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-osd", 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQCCRB1moF6vBRAA1uLHjPig3hzx+RNVVvk26Q==", 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-osd" 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd", 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQCCRB1m7WSvBRAA0u9vk22p2Pjt2Sc1f85gXg==", 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQCCRB1mv2uvBRAA65AI7dWdSekdgi5mhGFVhQ==", 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rgw", 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQCCRB1mEHOvBRAAKisrY/7THeReyJ2wKtX+Dg==", 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBiRB1mw7fVAxAAugywxKnWYMnJYE7dZdjnsw==", 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.255 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQBhRB1mABxSOxAAoEB3jK+dAm0BhB+SFLUyxA==", 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13f406c920 msgr2=0x7f13f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13f406c920 0x7f13f406ede0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f140c036000 tx=0x7f140c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 msgr2=0x7f14100cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 0x7f14100cd160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f140000f8e0 tx=0x7f140000fda0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 shutdown_connections 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13f406c920 0x7f13f406ede0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f14100cd6a0 0x7f14100cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14100ca7e0 0x7f14100cd160 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 --2- 172.21.15.46:0/2176239784 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13fc0039e0 0x7f14100cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 >> 172.21.15.46:0/2176239784 conn(0x7f141009d780 msgr2=0x7f141015d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 shutdown_connections 2024-04-15T15:16:55.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.240+0000 7f141719f640 1 -- 172.21.15.46:0/2176239784 wait complete. 2024-04-15T15:16:55.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-15T15:16:55.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.363+0000 7f2d793a8640 1 Processor -- start 2024-04-15T15:16:55.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.363+0000 7f2d793a8640 1 -- start start 2024-04-15T15:16:55.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d793a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 0x7f2d740d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d793a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d7415db40 con 0x7f2d74072d80 2024-04-15T15:16:55.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d793a8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d74072170 con 0x7f2d740d0bd0 2024-04-15T15:16:55.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d793a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2d7415cfc0 con 0x7f2d74071d10 2024-04-15T15:16:55.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d72ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d74072d80 0x7f2d740af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35482/0 (socket says 172.21.15.46:35482) 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d72ffd640 1 -- 172.21.15.46:0/1260216637 learned_addr learned my addr 172.21.15.46:0/1260216637 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3932574427 0 0) 0x7f2d7415db40 con 0x7f2d74072d80 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d60003520 con 0x7f2d74072d80 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 0x7f2d740d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3232517742 0 0) 0x7f2d60003520 con 0x7f2d74072d80 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 msgr2=0x7f2d740d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 0x7f2d740d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 >> v1:172.21.15.78:6789/0 conn(0x7f2d740d0bd0 legacy=0x7f2d7406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7415d2a0 con 0x7f2d74072d80 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f2d64001670 con 0x7f2d74072d80 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 >> v1:172.21.15.46:6789/0 conn(0x7f2d74072d80 legacy=0x7f2d740af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 0x7f2d600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.364+0000 7f2d727fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d60004d40 0x7f2d600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7415cfc0 con 0x7f2d60003f60 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6000bb50 con 0x7f2d60004d40 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6000bd50 con 0x7f2d74071d10 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d72ffd640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 0x7f2d600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d737fe640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d60004d40 0x7f2d600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d6a7fc640 1 -- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 msgr2=0x7f2d600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d74071d10 0x7f2d600039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d6a7fc640 1 -- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d60004d40 msgr2=0x7f2d600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d60004d40 0x7f2d600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.365+0000 7f2d6a7fc640 1 -- 172.21.15.46:0/1260216637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d60003520 con 0x7f2d60003f60 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d60004800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009ac0 tx=0x7f2d60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35f9051007c93b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c039070 con 0x7f2d60003f60 2024-04-15T15:16:55.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5c02c6b0 con 0x7f2d60003f60 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d727fc640 1 -- 172.21.15.46:0/1260216637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c009690 con 0x7f2d60003f60 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d793a8640 1 -- 172.21.15.46:0/1260216637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 msgr2=0x7f2d60004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d793a8640 1 --2- 172.21.15.46:0/1260216637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d60004800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009ac0 tx=0x7f2d60005830 comp rx=0 tx=0).stop 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d737fe640 1 -- 172.21.15.46:0/1260216637 reap_dead start 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d793a8640 1 -- 172.21.15.46:0/1260216637 shutdown_connections 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d793a8640 1 -- 172.21.15.46:0/1260216637 >> 172.21.15.46:0/1260216637 conn(0x7f2d7406be00 msgr2=0x7f2d7406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d793a8640 1 -- 172.21.15.46:0/1260216637 shutdown_connections 2024-04-15T15:16:55.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.366+0000 7f2d793a8640 1 -- 172.21.15.46:0/1260216637 wait complete. 2024-04-15T15:16:55.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.368+0000 7f2d793a8640 1 Processor -- start 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.368+0000 7f2d793a8640 1 -- start start 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.368+0000 7f2d793a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d7409dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.368+0000 7f2d793a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d741f5020 0x7f2d741f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.368+0000 7f2d793a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d741f5a90 0x7f2d741f1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d793a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d740afaf0 con 0x7f2d60003f60 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d793a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7415d860 con 0x7f2d741f5020 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d793a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2d7415cb90 con 0x7f2d741f5a90 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d7409dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d7409dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47608/0 (socket says 172.21.15.46:47608) 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 -- 172.21.15.46:0/83195789 learned_addr learned my addr 172.21.15.46:0/83195789 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d737fe640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d741f5a90 0x7f2d741f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d741f5020 0x7f2d741f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 -- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d741f5a90 msgr2=0x7f2d741f1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d741f5a90 0x7f2d741f1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 -- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d741f5020 msgr2=0x7f2d741f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d741f5020 0x7f2d741f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 -- 172.21.15.46:0/83195789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d74148600 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d72ffd640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d7409dde0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2d640072f0 tx=0x7f2d640077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400f8e0 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d64018070 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d741488e0 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6400fcc0 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.369+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d74148ea0 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.370+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d740689e0 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.371+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d6401d460 con 0x7f2d60003f60 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.371+0000 7f2d6bfff640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d5406c920 0x7f2d5406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.371+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f2d6408bad0 con 0x7f2d60003f60 2024-04-15T15:16:55.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.371+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d5406c920 0x7f2d5406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.372+0000 7f2d6a7fc640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d5406c920 0x7f2d5406ede0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009ac0 tx=0x7f2d5c030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.376+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d6400fe60 con 0x7f2d60003f60 2024-04-15T15:16:55.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.632+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f2d740cc730 con 0x7f2d60003f60 2024-04-15T15:16:55.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.634+0000 7f2d6bfff640 1 -- 172.21.15.46:0/83195789 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f2d64059a70 con 0x7f2d60003f60 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d5406c920 msgr2=0x7f2d5406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d5406c920 0x7f2d5406ede0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009ac0 tx=0x7f2d5c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 msgr2=0x7f2d7409dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d7409dde0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2d640072f0 tx=0x7f2d640077b0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 shutdown_connections 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d5406c920 0x7f2d5406ede0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2d741f5a90 0x7f2d741f1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2d741f5020 0x7f2d741f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 --2- 172.21.15.46:0/83195789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d60003f60 0x7f2d7409dde0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 >> 172.21.15.46:0/83195789 conn(0x7f2d7406be00 msgr2=0x7f2d74160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 shutdown_connections 2024-04-15T15:16:55.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.636+0000 7f2d793a8640 1 -- 172.21.15.46:0/83195789 wait complete. 2024-04-15T15:16:55.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-15T15:16:55.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.753+0000 7fa41a463640 1 Processor -- start 2024-04-15T15:16:55.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.753+0000 7fa41a463640 1 -- start start 2024-04-15T15:16:55.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa41a463640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 0x7fa4140d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa41a463640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa41415eb70 con 0x7fa4140d15e0 2024-04-15T15:16:55.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa41a463640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4140d3c80 con 0x7fa41415e790 2024-04-15T15:16:55.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa41a463640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa4140d3790 con 0x7fa4140d5150 2024-04-15T15:16:55.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa40bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa4140d15e0 0x7fa4140d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35488/0 (socket says 172.21.15.46:35488) 2024-04-15T15:16:55.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa40bfff640 1 -- 172.21.15.46:0/3737012424 learned_addr learned my addr 172.21.15.46:0/3737012424 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa418a00640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 0x7fa4140d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 527336283 0 0) 0x7fa41415eb70 con 0x7fa4140d15e0 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.754+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa400003520 con 0x7fa4140d15e0 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800659876 0 0) 0x7fa4140d3c80 con 0x7fa41415e790 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa41415eb70 con 0x7fa41415e790 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 883252937 0 0) 0x7fa400003520 con 0x7fa4140d15e0 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 msgr2=0x7fa4140d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 0x7fa4140d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 >> v1:172.21.15.78:6789/0 conn(0x7fa41415e790 legacy=0x7fa4140b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4140d3f60 con 0x7fa4140d15e0 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fa3fc001670 con 0x7fa4140d15e0 2024-04-15T15:16:55.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 >> v1:172.21.15.46:6789/0 conn(0x7fa4140d15e0 legacy=0x7fa4140d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400003b70 0x7fa400003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 0x7fa400004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa400004d40 0x7fa4000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4140d3790 con 0x7fa400004d40 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa40000bbd0 con 0x7fa400003b70 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa40000bdd0 con 0x7fa4140d5150 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa418a00640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa400004d40 0x7fa4000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa413fff640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400003b70 0x7fa400003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa40bfff640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 0x7fa400004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa418a00640 1 -- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 msgr2=0x7fa400004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa418a00640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4140d5150 0x7fa400004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa418a00640 1 -- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400003b70 msgr2=0x7fa400003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa418a00640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400003b70 0x7fa400003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.755+0000 7fa418a00640 1 -- 172.21.15.46:0/3737012424 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa400003520 con 0x7fa400004d40 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.756+0000 7fa418a00640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa400004d40 0x7fa4000051c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009fc0 tx=0x7fa400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161445a64180adee server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.756+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c039070 con 0x7fa400004d40 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.756+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40c02cc20 con 0x7fa400004d40 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.756+0000 7fa4137fe640 1 -- 172.21.15.46:0/3737012424 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c009c10 con 0x7fa400004d40 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa41a463640 1 -- 172.21.15.46:0/3737012424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa400004d40 msgr2=0x7fa4000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa41a463640 1 --2- 172.21.15.46:0/3737012424 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa400004d40 0x7fa4000051c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009fc0 tx=0x7fa400005830 comp rx=0 tx=0).stop 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa418a00640 1 -- 172.21.15.46:0/3737012424 reap_dead start 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa41a463640 1 -- 172.21.15.46:0/3737012424 shutdown_connections 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa41a463640 1 -- 172.21.15.46:0/3737012424 >> 172.21.15.46:0/3737012424 conn(0x7fa414067b40 msgr2=0x7fa414067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa41a463640 1 -- 172.21.15.46:0/3737012424 shutdown_connections 2024-04-15T15:16:55.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.757+0000 7fa41a463640 1 -- 172.21.15.46:0/3737012424 wait complete. 2024-04-15T15:16:55.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.758+0000 7fa41a463640 1 Processor -- start 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.758+0000 7fa41a463640 1 -- start start 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa41a463640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400004d40 0x7fa414155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa41a463640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4141f4ff0 0x7fa4140cab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa41a463640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 0x7fa4140cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa41a463640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa4140b7db0 con 0x7fa4140cb060 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa41a463640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa4140d45b0 con 0x7fa400004d40 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa41a463640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa4140d1a90 con 0x7fa4141f4ff0 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 0x7fa4140cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 0x7fa4140cd900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47620/0 (socket says 172.21.15.46:47620) 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 -- 172.21.15.46:0/3674628578 learned_addr learned my addr 172.21.15.46:0/3674628578 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa40bfff640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4141f4ff0 0x7fa4140cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa413fff640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400004d40 0x7fa414155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 -- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4141f4ff0 msgr2=0x7fa4140cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4141f4ff0 0x7fa4140cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 -- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400004d40 msgr2=0x7fa414155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400004d40 0x7fa414155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 -- 172.21.15.46:0/3674628578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4140cc860 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.759+0000 7fa418a00640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 0x7fa4140cd900 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa40c02c690 tx=0x7fa40c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.760+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c039070 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.760+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40c044070 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.760+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4140ccb40 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.760+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c03e810 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.760+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4140cd100 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.761+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4140689e0 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.761+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa40c005d10 con 0x7fa4140cb060 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.762+0000 7fa4117fa640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3f406c920 0x7fa3f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.762+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fa40c0ad9d0 con 0x7fa4140cb060 2024-04-15T15:16:55.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.762+0000 7fa413fff640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3f406c920 0x7fa3f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.763+0000 7fa413fff640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3f406c920 0x7fa3f406ede0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4040010e0 tx=0x7fa404009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:55.766+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa40c04d020 con 0x7fa4140cb060 2024-04-15T15:16:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.024+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fa4140b7a80 con 0x7fa4140cb060 2024-04-15T15:16:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.025+0000 7fa4117fa640 1 -- 172.21.15.46:0/3674628578 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fa40c07ba00 con 0x7fa4140cb060 2024-04-15T15:16:56.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3f406c920 msgr2=0x7fa3f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3f406c920 0x7fa3f406ede0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4040010e0 tx=0x7fa404009040 comp rx=0 tx=0).stop 2024-04-15T15:16:56.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 msgr2=0x7fa4140cd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 0x7fa4140cd900 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa40c02c690 tx=0x7fa40c009750 comp rx=0 tx=0).stop 2024-04-15T15:16:56.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 shutdown_connections 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa3f406c920 0x7fa3f406ede0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4140cb060 0x7fa4140cd900 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa4141f4ff0 0x7fa4140cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 --2- 172.21.15.46:0/3674628578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa400004d40 0x7fa414155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 >> 172.21.15.46:0/3674628578 conn(0x7fa414067b40 msgr2=0x7fa4140d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 shutdown_connections 2024-04-15T15:16:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.027+0000 7fa41a463640 1 -- 172.21.15.46:0/3674628578 wait complete. 2024-04-15T15:16:56.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-15T15:16:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.144+0000 7fada1b00640 1 Processor -- start 2024-04-15T15:16:56.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.144+0000 7fada1b00640 1 -- start start 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fada1b00640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad9c0d1a10 0x7fad9c0b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fada1b00640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d08a0 0x7fad9c0b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fada1b00640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad9c15ba70 con 0x7fad9c15b690 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fada1b00640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fad9c0d0d50 con 0x7fad9c0d1a10 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fada1b00640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fad9c0ce5b0 con 0x7fad9c0d08a0 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad93fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fad9c0d1a10 0x7fad9c0b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad9b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fad9c15b690 0x7fad9c0d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35504/0 (socket says 172.21.15.46:35504) 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad9b7fe640 1 -- 172.21.15.46:0/3894409142 learned_addr learned my addr 172.21.15.46:0/3894409142 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:56.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad9bfff640 1 --2- 172.21.15.46:0/3894409142 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d08a0 0x7fad9c0b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad93fff640 1 -- 172.21.15.46:0/3894409142 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d08a0 msgr2=0x7fad9c0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad93fff640 1 --2- 172.21.15.46:0/3894409142 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d08a0 0x7fad9c0b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad93fff640 1 -- 172.21.15.46:0/3894409142 >> v1:172.21.15.46:6789/0 conn(0x7fad9c15b690 legacy=0x7fad9c0d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad9affd640 1 -- 172.21.15.46:0/3894409142 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833274998 0 0) 0x7fad8c001110 con 0x7fad9c15b690 2024-04-15T15:16:56.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.145+0000 7fad93fff640 1 -- 172.21.15.46:0/3894409142 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9c0ce890 con 0x7fad9c0d1a10 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.146+0000 7fad93fff640 1 --2- 172.21.15.46:0/3894409142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c0d1a10 0x7fad9c0b6ff0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fad840097d0 tx=0x7fad8402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846bae2ae1426b28 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.146+0000 7fad9affd640 1 -- 172.21.15.46:0/3894409142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84035020 con 0x7fad9c0d1a10 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.146+0000 7fad9affd640 1 -- 172.21.15.46:0/3894409142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad84031800 con 0x7fad9c0d1a10 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.146+0000 7fad9affd640 1 -- 172.21.15.46:0/3894409142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad84031bd0 con 0x7fad9c0d1a10 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 -- 172.21.15.46:0/3894409142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c0d1a10 msgr2=0x7fad9c0b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 --2- 172.21.15.46:0/3894409142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c0d1a10 0x7fad9c0b6ff0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fad840097d0 tx=0x7fad8402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 -- 172.21.15.46:0/3894409142 shutdown_connections 2024-04-15T15:16:56.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 --2- 172.21.15.46:0/3894409142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c0d1a10 0x7fad9c0b6ff0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 --2- 172.21.15.46:0/3894409142 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d08a0 0x7fad9c0b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 -- 172.21.15.46:0/3894409142 >> 172.21.15.46:0/3894409142 conn(0x7fad9c09a9d0 msgr2=0x7fad9c0b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 -- 172.21.15.46:0/3894409142 shutdown_connections 2024-04-15T15:16:56.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.147+0000 7fada1b00640 1 -- 172.21.15.46:0/3894409142 wait complete. 2024-04-15T15:16:56.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.148+0000 7fada1b00640 1 Processor -- start 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.148+0000 7fada1b00640 1 -- start start 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad9c0d08a0 0x7fad9c0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d1a10 0x7fad9c1ed650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 0x7fad9c15df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c0b1ba0 con 0x7fad9c15b690 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c0d1010 con 0x7fad9c0d08a0 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fad9c0b1d20 con 0x7fad9c0d1a10 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 0x7fad9c15df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 0x7fad9c15df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47646/0 (socket says 172.21.15.46:47646) 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 -- 172.21.15.46:0/3594669312 learned_addr learned my addr 172.21.15.46:0/3594669312 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad93fff640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d1a10 0x7fad9c1ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9b7fe640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad9c0d08a0 0x7fad9c0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 -- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d1a10 msgr2=0x7fad9c1ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d1a10 0x7fad9c1ed650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 -- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad9c0d08a0 msgr2=0x7fad9c0ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad9c0d08a0 0x7fad9c0ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 -- 172.21.15.46:0/3594669312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9c15e480 con 0x7fad9c15b690 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9b7fe640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad9c0d08a0 0x7fad9c0ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad9bfff640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 0x7fad9c15df40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fad9400efc0 tx=0x7fad9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94018020 con 0x7fad9c15b690 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad94014480 con 0x7fad9c15b690 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9c0ae980 con 0x7fad9c15b690 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.149+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad94014870 con 0x7fad9c15b690 2024-04-15T15:16:56.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.150+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad9c0aef40 con 0x7fad9c15b690 2024-04-15T15:16:56.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.151+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad9c0690f0 con 0x7fad9c15b690 2024-04-15T15:16:56.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.151+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fad94005d10 con 0x7fad9c15b690 2024-04-15T15:16:56.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.152+0000 7fad98ff9640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad7c06c920 0x7fad7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.152+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fad9408c3c0 con 0x7fad9c15b690 2024-04-15T15:16:56.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.152+0000 7fad9b7fe640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad7c06c920 0x7fad7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.152+0000 7fad9b7fe640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad7c06c920 0x7fad7c06ede0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fad8c000f60 tx=0x7fad8c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.156+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad94022070 con 0x7fad9c15b690 2024-04-15T15:16:56.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.410+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fad9c06f180 con 0x7fad9c15b690 2024-04-15T15:16:56.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.412+0000 7fad98ff9640 1 -- 172.21.15.46:0/3594669312 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7fad94010040 con 0x7fad9c15b690 2024-04-15T15:16:56.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.414+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad7c06c920 msgr2=0x7fad7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.414+0000 7fada1b00640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad7c06c920 0x7fad7c06ede0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fad8c000f60 tx=0x7fad8c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.414+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 msgr2=0x7fad9c15df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.414+0000 7fada1b00640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 0x7fad9c15df40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fad9400efc0 tx=0x7fad9400c490 comp rx=0 tx=0).stop 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 shutdown_connections 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fad7c06c920 0x7fad7c06ede0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad9c15b690 0x7fad9c15df40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fad9c0d1a10 0x7fad9c1ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 --2- 172.21.15.46:0/3594669312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad9c0d08a0 0x7fad9c0ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 >> 172.21.15.46:0/3594669312 conn(0x7fad9c09a9d0 msgr2=0x7fad9c158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 shutdown_connections 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.415+0000 7fada1b00640 1 -- 172.21.15.46:0/3594669312 wait complete. 2024-04-15T15:16:56.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-15T15:16:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.531+0000 7f5650835640 1 Processor -- start 2024-04-15T15:16:56.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.531+0000 7f5650835640 1 -- start start 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5650835640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f564c15e790 0x7f564c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5650835640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f564c0d1610 0x7f564c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5650835640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f564c0d2480 con 0x7f564c0cfd20 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5650835640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f564c0d2db0 con 0x7f564c15e790 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5650835640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f564c0d2760 con 0x7f564c0d1610 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f564a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f564c15e790 0x7f564c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5641d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f564c0cfd20 0x7f564c0d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35520/0 (socket says 172.21.15.46:35520) 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f5641d9b640 1 -- 172.21.15.46:0/2571930618 learned_addr learned my addr 172.21.15.46:0/2571930618 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f564ad9d640 1 --2- 172.21.15.46:0/2571930618 >> v2:172.21.15.78:3300/0 conn(0x7f564c0d1610 0x7f564c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f564a59c640 1 -- 172.21.15.46:0/2571930618 >> v2:172.21.15.78:3300/0 conn(0x7f564c0d1610 msgr2=0x7f564c0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f564a59c640 1 --2- 172.21.15.46:0/2571930618 >> v2:172.21.15.78:3300/0 conn(0x7f564c0d1610 0x7f564c0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f564a59c640 1 -- 172.21.15.46:0/2571930618 >> v1:172.21.15.46:6789/0 conn(0x7f564c0cfd20 legacy=0x7f564c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.532+0000 7f564a59c640 1 -- 172.21.15.46:0/2571930618 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c0d2a40 con 0x7f564c15e790 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.533+0000 7f564a59c640 1 --2- 172.21.15.46:0/2571930618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c0d3240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f563c0097a0 tx=0x7f563c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16c650ea10f5b0e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.533+0000 7f5649d9b640 1 -- 172.21.15.46:0/2571930618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c035020 con 0x7f564c15e790 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.533+0000 7f5649d9b640 1 -- 172.21.15.46:0/2571930618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f563c031830 con 0x7f564c15e790 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.533+0000 7f5649d9b640 1 -- 172.21.15.46:0/2571930618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f563c031c00 con 0x7f564c15e790 2024-04-15T15:16:56.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.533+0000 7f5650835640 1 -- 172.21.15.46:0/2571930618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 msgr2=0x7f564c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 --2- 172.21.15.46:0/2571930618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c0d3240 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f563c0097a0 tx=0x7f563c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 -- 172.21.15.46:0/2571930618 shutdown_connections 2024-04-15T15:16:56.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 --2- 172.21.15.46:0/2571930618 >> v2:172.21.15.78:3300/0 conn(0x7f564c0d1610 0x7f564c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 --2- 172.21.15.46:0/2571930618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c0d3240 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 -- 172.21.15.46:0/2571930618 >> 172.21.15.46:0/2571930618 conn(0x7f564c08bf60 msgr2=0x7f564c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 -- 172.21.15.46:0/2571930618 shutdown_connections 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.534+0000 7f5650835640 1 -- 172.21.15.46:0/2571930618 wait complete. 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.535+0000 7f5650835640 1 Processor -- start 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.535+0000 7f5650835640 1 -- start start 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5650835640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f564c0cfd20 0x7f564c154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5650835640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f564c0d1610 0x7f564c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5650835640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5650835640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f564c15ecf0 con 0x7f564c15e790 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5650835640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f564c15eb70 con 0x7f564c0d1610 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5650835640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f564c0b74f0 con 0x7f564c0cfd20 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c06e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47666/0 (socket says 172.21.15.46:47666) 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 -- 172.21.15.46:0/1500417655 learned_addr learned my addr 172.21.15.46:0/1500417655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564a59c640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f564c0cfd20 0x7f564c154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f5641d9b640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f564c0d1610 0x7f564c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 -- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f564c0cfd20 msgr2=0x7f564c154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f564c0cfd20 0x7f564c154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 -- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f564c0d1610 msgr2=0x7f564c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f564c0d1610 0x7f564c06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 -- 172.21.15.46:0/1500417655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c155180 con 0x7f564c15e790 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.536+0000 7f564ad9d640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c06e1a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f564400efc0 tx=0x7f564400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.537+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644018020 con 0x7f564c15e790 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.537+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5644014480 con 0x7f564c15e790 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.537+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f564c0b46d0 con 0x7f564c15e790 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.537+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644014870 con 0x7f564c15e790 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.537+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f564c0b4c90 con 0x7f564c15e790 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.538+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f564c0689e0 con 0x7f564c15e790 2024-04-15T15:16:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.538+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5644005d10 con 0x7f564c15e790 2024-04-15T15:16:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.539+0000 7f56437fe640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f562c06c8f0 0x7f562c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.539+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f564408c7d0 con 0x7f564c15e790 2024-04-15T15:16:56.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.539+0000 7f564a59c640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f562c06c8f0 0x7f562c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.539+0000 7f564a59c640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f562c06c8f0 0x7f562c06edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f563c036000 tx=0x7f563c03c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.543+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5644022030 con 0x7f564c15e790 2024-04-15T15:16:56.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.798+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f564c0d1ce0 con 0x7f564c15e790 2024-04-15T15:16:56.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.800+0000 7f56437fe640 1 -- 172.21.15.46:0/1500417655 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f5644010040 con 0x7f564c15e790 2024-04-15T15:16:56.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f562c06c8f0 msgr2=0x7f562c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f562c06c8f0 0x7f562c06edb0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f563c036000 tx=0x7f563c03c040 comp rx=0 tx=0).stop 2024-04-15T15:16:56.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 msgr2=0x7f564c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c06e1a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f564400efc0 tx=0x7f564400c490 comp rx=0 tx=0).stop 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 shutdown_connections 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f562c06c8f0 0x7f562c06edb0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f564c15e790 0x7f564c06e1a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f564c0d1610 0x7f564c06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 --2- 172.21.15.46:0/1500417655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f564c0cfd20 0x7f564c154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 >> 172.21.15.46:0/1500417655 conn(0x7f564c08bf60 msgr2=0x7f564c0cafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 shutdown_connections 2024-04-15T15:16:56.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.802+0000 7f5650835640 1 -- 172.21.15.46:0/1500417655 wait complete. 2024-04-15T15:16:56.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-15T15:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.919+0000 7f2da53f2640 1 Processor -- start 2024-04-15T15:16:56.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.919+0000 7f2da53f2640 1 -- start start 2024-04-15T15:16:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2da53f2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2da01602d0 0x7f2da00cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2da53f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da009afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2da53f2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da015dbd0 con 0x7f2da01606b0 2024-04-15T15:16:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2da53f2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2da00cd6a0 con 0x7f2da01602d0 2024-04-15T15:16:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2da53f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2da00cd980 con 0x7f2da009ab40 2024-04-15T15:16:56.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2d9e7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2da01606b0 0x7f2da00d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35532/0 (socket says 172.21.15.46:35532) 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2d9e7fc640 1 -- 172.21.15.46:0/2557945702 learned_addr learned my addr 172.21.15.46:0/2557945702 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2d9effd640 1 --2- 172.21.15.46:0/2557945702 >> v2:172.21.15.78:3300/0 conn(0x7f2da01602d0 0x7f2da00cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2d9f7fe640 1 --2- 172.21.15.46:0/2557945702 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da009afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2d9dffb640 1 -- 172.21.15.46:0/2557945702 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 317931425 0 0) 0x7f2da015dbd0 con 0x7f2da01606b0 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.920+0000 7f2d9dffb640 1 -- 172.21.15.46:0/2557945702 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d88003520 con 0x7f2da01606b0 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9f7fe640 1 -- 172.21.15.46:0/2557945702 >> v2:172.21.15.78:3300/0 conn(0x7f2da01602d0 msgr2=0x7f2da00cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9f7fe640 1 --2- 172.21.15.46:0/2557945702 >> v2:172.21.15.78:3300/0 conn(0x7f2da01602d0 0x7f2da00cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9f7fe640 1 -- 172.21.15.46:0/2557945702 >> v1:172.21.15.46:6789/0 conn(0x7f2da01606b0 legacy=0x7f2da00d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9dffb640 1 -- 172.21.15.46:0/2557945702 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3293872756 0 0) 0x7f2d88003520 con 0x7f2da01606b0 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9f7fe640 1 -- 172.21.15.46:0/2557945702 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da00cdc60 con 0x7f2da009ab40 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9effd640 1 --2- 172.21.15.46:0/2557945702 >> v2:172.21.15.78:3300/0 conn(0x7f2da01602d0 0x7f2da00cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.921+0000 7f2d9f7fe640 1 --2- 172.21.15.46:0/2557945702 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da009afd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097a0 tx=0x7f2d9802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23ad690c8ce26b02 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2d9dffb640 1 -- 172.21.15.46:0/2557945702 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98037070 con 0x7f2da009ab40 2024-04-15T15:16:56.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2d9dffb640 1 -- 172.21.15.46:0/2557945702 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98030510 con 0x7f2da009ab40 2024-04-15T15:16:56.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2d9dffb640 1 -- 172.21.15.46:0/2557945702 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94001670 con 0x7f2da009ab40 2024-04-15T15:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 -- 172.21.15.46:0/2557945702 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 msgr2=0x7f2da009afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2557945702 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da009afd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2d980097a0 tx=0x7f2d9802b790 comp rx=0 tx=0).stop 2024-04-15T15:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 -- 172.21.15.46:0/2557945702 shutdown_connections 2024-04-15T15:16:56.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2557945702 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da009afd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2557945702 >> v2:172.21.15.78:3300/0 conn(0x7f2da01602d0 0x7f2da00cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 -- 172.21.15.46:0/2557945702 >> 172.21.15.46:0/2557945702 conn(0x7f2da00d2460 msgr2=0x7f2da00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 -- 172.21.15.46:0/2557945702 shutdown_connections 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.922+0000 7f2da53f2640 1 -- 172.21.15.46:0/2557945702 wait complete. 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 Processor -- start 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 -- start start 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da006caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 0x7f2da01589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da01606b0 0x7f2da00b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2da0160b10 con 0x7f2da01602d0 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2da00ce220 con 0x7f2da01606b0 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2da53f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2da009b510 con 0x7f2da009ab40 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.924+0000 7f2d9e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 0x7f2da01589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 0x7f2da01589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47682/0 (socket says 172.21.15.46:47682) 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 -- 172.21.15.46:0/2207340878 learned_addr learned my addr 172.21.15.46:0/2207340878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9effd640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da006caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9f7fe640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da01606b0 0x7f2da00b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 -- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 msgr2=0x7f2da006caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da006caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 -- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da01606b0 msgr2=0x7f2da00b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da01606b0 0x7f2da00b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 -- 172.21.15.46:0/2207340878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da00c5220 con 0x7f2da01602d0 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d9e7fc640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 0x7f2da01589d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2d940017e0 tx=0x7f2d9400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94005ad0 con 0x7f2da01602d0 2024-04-15T15:16:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94005c90 con 0x7f2da01602d0 2024-04-15T15:16:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da00c5500 con 0x7f2da01602d0 2024-04-15T15:16:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d940072b0 con 0x7f2da01602d0 2024-04-15T15:16:56.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.925+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da01f7cc0 con 0x7f2da01602d0 2024-04-15T15:16:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.926+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da00689e0 con 0x7f2da01602d0 2024-04-15T15:16:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.927+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d94007450 con 0x7f2da01602d0 2024-04-15T15:16:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.927+0000 7f2d877fe640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d8c06c920 0x7f2d8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.927+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f2d9408b730 con 0x7f2da01602d0 2024-04-15T15:16:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.928+0000 7f2d9effd640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d8c06c920 0x7f2d8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.928+0000 7f2d9effd640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d8c06c920 0x7f2d8c06ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2da00ce940 tx=0x7f2d9000b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:56.932+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d94010300 con 0x7f2da01602d0 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.190+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f2da00c56d0 con 0x7f2da01602d0 2024-04-15T15:16:57.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.193+0000 7f2d877fe640 1 -- 172.21.15.46:0/2207340878 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f2d94059db0 con 0x7f2da01602d0 2024-04-15T15:16:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d8c06c920 msgr2=0x7f2d8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d8c06c920 0x7f2d8c06ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2da00ce940 tx=0x7f2d9000b040 comp rx=0 tx=0).stop 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 msgr2=0x7f2da01589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 0x7f2da01589d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2d940017e0 tx=0x7f2d9400cbe0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 shutdown_connections 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2d8c06c920 0x7f2d8c06ede0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2da01606b0 0x7f2da00b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.194+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da01602d0 0x7f2da01589d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.195+0000 7f2da53f2640 1 --2- 172.21.15.46:0/2207340878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2da009ab40 0x7f2da006caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.195+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 >> 172.21.15.46:0/2207340878 conn(0x7f2da00d2460 msgr2=0x7f2da00b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.195+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 shutdown_connections 2024-04-15T15:16:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.195+0000 7f2da53f2640 1 -- 172.21.15.46:0/2207340878 wait complete. 2024-04-15T15:16:57.208 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-15T15:16:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.311+0000 7ff6eb9c8640 1 Processor -- start 2024-04-15T15:16:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.311+0000 7ff6eb9c8640 1 -- start start 2024-04-15T15:16:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6eb9c8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6e40d15e0 0x7ff6e40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6eb9c8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6e415eb70 con 0x7ff6e40d4a40 2024-04-15T15:16:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6eb9c8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6e40d3c80 con 0x7ff6e415e790 2024-04-15T15:16:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6eb9c8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff6e40d3790 con 0x7ff6e40d15e0 2024-04-15T15:16:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6e8f63640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff6e40d15e0 0x7ff6e40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6e9f65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff6e40d4a40 0x7ff6e40d29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35540/0 (socket says 172.21.15.46:35540) 2024-04-15T15:16:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.312+0000 7ff6e9f65640 1 -- 172.21.15.46:0/191821637 learned_addr learned my addr 172.21.15.46:0/191821637 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e8f63640 1 -- 172.21.15.46:0/191821637 >> v1:172.21.15.78:6789/0 conn(0x7ff6e415e790 legacy=0x7ff6e40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e8f63640 1 -- 172.21.15.46:0/191821637 >> v1:172.21.15.46:6789/0 conn(0x7ff6e40d4a40 legacy=0x7ff6e40d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e8f63640 1 -- 172.21.15.46:0/191821637 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e40d3f60 con 0x7ff6e40d15e0 2024-04-15T15:16:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e8f63640 1 --2- 172.21.15.46:0/191821637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e40d15e0 0x7ff6e40d1a90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80097a0 tx=0x7ff6d802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa5cccb5dbcbacf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e3fff640 1 -- 172.21.15.46:0/191821637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8038070 con 0x7ff6e40d15e0 2024-04-15T15:16:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e3fff640 1 -- 172.21.15.46:0/191821637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d8031700 con 0x7ff6e40d15e0 2024-04-15T15:16:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.313+0000 7ff6e3fff640 1 -- 172.21.15.46:0/191821637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8031ad0 con 0x7ff6e40d15e0 2024-04-15T15:16:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/191821637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e40d15e0 msgr2=0x7ff6e40d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/191821637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e40d15e0 0x7ff6e40d1a90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff6d80097a0 tx=0x7ff6d802b7a0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/191821637 shutdown_connections 2024-04-15T15:16:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/191821637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e40d15e0 0x7ff6e40d1a90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/191821637 >> 172.21.15.46:0/191821637 conn(0x7ff6e4067b40 msgr2=0x7ff6e4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/191821637 shutdown_connections 2024-04-15T15:16:57.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.314+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/191821637 wait complete. 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.315+0000 7ff6eb9c8640 1 Processor -- start 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.315+0000 7ff6eb9c8640 1 -- start start 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6eb9c8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6e40d15e0 0x7ff6e41498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6eb9c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e40d4a40 0x7ff6e41613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6eb9c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 0x7ff6e40c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6eb9c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e40b7db0 con 0x7ff6e415e790 2024-04-15T15:16:57.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6eb9c8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e40d45b0 con 0x7ff6e40d4a40 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6eb9c8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e40d30d0 con 0x7ff6e40d15e0 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 0x7ff6e40c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 0x7ff6e40c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47694/0 (socket says 172.21.15.46:47694) 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 -- 172.21.15.46:0/13946881 learned_addr learned my addr 172.21.15.46:0/13946881 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e8f63640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e40d4a40 0x7ff6e41613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9764640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6e40d15e0 0x7ff6e41498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 -- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6e40d15e0 msgr2=0x7ff6e41498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6e40d15e0 0x7ff6e41498d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 -- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e40d4a40 msgr2=0x7ff6e41613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e40d4a40 0x7ff6e41613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.316+0000 7ff6e9f65640 1 -- 172.21.15.46:0/13946881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e415ee10 con 0x7ff6e415e790 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.317+0000 7ff6e9f65640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 0x7ff6e40c9ab0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400cad0 tx=0x7ff6d400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.317+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4017070 con 0x7ff6e415e790 2024-04-15T15:16:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.317+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6d400ab90 con 0x7ff6e415e790 2024-04-15T15:16:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.317+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4012400 con 0x7ff6e415e790 2024-04-15T15:16:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.317+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e40c8ee0 con 0x7ff6e415e790 2024-04-15T15:16:57.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.317+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e40c9720 con 0x7ff6e415e790 2024-04-15T15:16:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.318+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6e415efe0 con 0x7ff6e415e790 2024-04-15T15:16:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.318+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff6d401a070 con 0x7ff6e415e790 2024-04-15T15:16:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.319+0000 7ff6e1ffb640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff6d006c8f0 0x7ff6d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.319+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7ff6d408b610 con 0x7ff6e415e790 2024-04-15T15:16:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.319+0000 7ff6e9764640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff6d006c8f0 0x7ff6d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.320+0000 7ff6e9764640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff6d006c8f0 0x7ff6d006edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc006fd0 tx=0x7ff6dc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.323+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff6e40c8ee0 con 0x7ff6e415e790 2024-04-15T15:16:57.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.583+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7ff6e40689e0 con 0x7ff6e415e790 2024-04-15T15:16:57.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.585+0000 7ff6e1ffb640 1 -- 172.21.15.46:0/13946881 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7ff6d4059c00 con 0x7ff6e415e790 2024-04-15T15:16:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff6d006c8f0 msgr2=0x7ff6d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff6d006c8f0 0x7ff6d006edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc006fd0 tx=0x7ff6dc008040 comp rx=0 tx=0).stop 2024-04-15T15:16:57.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 msgr2=0x7ff6e40c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 0x7ff6e40c9ab0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400cad0 tx=0x7ff6d400ced0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 shutdown_connections 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff6d006c8f0 0x7ff6d006edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6e415e790 0x7ff6e40c9ab0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6e40d4a40 0x7ff6e41613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 --2- 172.21.15.46:0/13946881 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6e40d15e0 0x7ff6e41498d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 >> 172.21.15.46:0/13946881 conn(0x7ff6e4067b40 msgr2=0x7ff6e40b99e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 shutdown_connections 2024-04-15T15:16:57.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.587+0000 7ff6eb9c8640 1 -- 172.21.15.46:0/13946881 wait complete. 2024-04-15T15:16:57.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:57.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.703+0000 7f0e6bf59640 1 Processor -- start 2024-04-15T15:16:57.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.703+0000 7f0e6bf59640 1 -- start start 2024-04-15T15:16:57.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.704+0000 7f0e6bf59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 0x7f0e640d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.704+0000 7f0e6bf59640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e640d2480 con 0x7f0e6415e790 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.704+0000 7f0e6bf59640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e640d2760 con 0x7f0e640d1440 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.704+0000 7f0e6bf59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0e640d2db0 con 0x7f0e640d3240 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.704+0000 7f0e69cf5640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0e6415e790 0x7f0e640cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35550/0 (socket says 172.21.15.46:35550) 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.704+0000 7f0e69cf5640 1 -- 172.21.15.46:0/795976606 learned_addr learned my addr 172.21.15.46:0/795976606 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e694f4640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 0x7f0e640d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667124885 0 0) 0x7f0e640d2480 con 0x7f0e6415e790 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e4c003520 con 0x7f0e6415e790 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2414883173 0 0) 0x7f0e4c003520 con 0x7f0e6415e790 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 msgr2=0x7f0e640d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 0x7f0e640d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 >> v1:172.21.15.78:6789/0 conn(0x7f0e640d1440 legacy=0x7f0e640b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e640d2a40 con 0x7f0e6415e790 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f0e5c001670 con 0x7f0e6415e790 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 >> v1:172.21.15.46:6789/0 conn(0x7f0e6415e790 legacy=0x7f0e640cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c0039e0 0x7f0e4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 0x7f0e4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c004d40 0x7f0e4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0e640d2db0 con 0x7f0e4c004d40 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c00ba70 con 0x7f0e4c0039e0 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0e4c00bbf0 con 0x7f0e640d3240 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c004d40 0x7f0e4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e694f4640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 0x7f0e4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e69cf5640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c0039e0 0x7f0e4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 msgr2=0x7f0e4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640d3240 0x7f0e4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c0039e0 msgr2=0x7f0e4c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c0039e0 0x7f0e4c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/795976606 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c003520 con 0x7f0e4c004d40 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.705+0000 7f0e69cf5640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c0039e0 0x7f0e4c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.706+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c004d40 0x7f0e4c0051c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0e54001f60 tx=0x7f0e4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778be632a53e669d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.706+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54001b30 con 0x7f0e4c004d40 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.706+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5402d710 con 0x7f0e4c004d40 2024-04-15T15:16:57.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.706+0000 7f0e68cf3640 1 -- 172.21.15.46:0/795976606 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e5402db00 con 0x7f0e4c004d40 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6bf59640 1 -- 172.21.15.46:0/795976606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c004d40 msgr2=0x7f0e4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/795976606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e4c004d40 0x7f0e4c0051c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0e54001f60 tx=0x7f0e4c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/795976606 reap_dead start 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6bf59640 1 -- 172.21.15.46:0/795976606 shutdown_connections 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6bf59640 1 -- 172.21.15.46:0/795976606 >> 172.21.15.46:0/795976606 conn(0x7f0e6408bf60 msgr2=0x7f0e640d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6bf59640 1 -- 172.21.15.46:0/795976606 shutdown_connections 2024-04-15T15:16:57.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.707+0000 7f0e6bf59640 1 -- 172.21.15.46:0/795976606 wait complete. 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.708+0000 7f0e6bf59640 1 Processor -- start 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.708+0000 7f0e6bf59640 1 -- start start 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6bf59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c004d40 0x7f0e6415fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6bf59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640c9450 0x7f0e640cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6bf59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 0x7f0e640cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6bf59640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0e640d5180 con 0x7f0e640c98b0 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6bf59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0e640d0430 con 0x7f0e4c004d40 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6bf59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0e640d5300 con 0x7f0e640c9450 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 0x7f0e640cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 0x7f0e640cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47712/0 (socket says 172.21.15.46:47712) 2024-04-15T15:16:57.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/698207872 learned_addr learned my addr 172.21.15.46:0/698207872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e694f4640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640c9450 0x7f0e640cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e69cf5640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c004d40 0x7f0e6415fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640c9450 msgr2=0x7f0e640cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640c9450 0x7f0e640cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c004d40 msgr2=0x7f0e6415fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c004d40 0x7f0e6415fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 -- 172.21.15.46:0/698207872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6415ecf0 con 0x7f0e640c98b0 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.709+0000 7f0e6a4f6640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 0x7f0e640cdc50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0e54008fd0 tx=0x7f0e540015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.710+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54001b30 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.710+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e54043070 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.710+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6415efd0 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.710+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54003bb0 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.710+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6415fff0 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.711+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e6408e0a0 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.711+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e54005070 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.711+0000 7f0e627fc640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0e5006c920 0x7f0e5006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.712+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f0e540ad620 con 0x7f0e640c98b0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.712+0000 7f0e69cf5640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0e5006c920 0x7f0e5006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.712+0000 7f0e69cf5640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0e5006c920 0x7f0e5006ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0017e0 tx=0x7f0e5c007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.716+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e6408e0a0 con 0x7f0e640c98b0 2024-04-15T15:16:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.902+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e640689e0 con 0x7f0e640c98b0 2024-04-15T15:16:57.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.903+0000 7f0e627fc640 1 -- 172.21.15.46:0/698207872 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13116 (secure 0 0 0) 0x7f0e5407bb90 con 0x7f0e640c98b0 2024-04-15T15:16:57.905 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:57.905 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":35,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:16:54.846067+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:57.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0e5006c920 msgr2=0x7f0e5006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0e5006c920 0x7f0e5006ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c0017e0 tx=0x7f0e5c007040 comp rx=0 tx=0).stop 2024-04-15T15:16:57.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 msgr2=0x7f0e640cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 0x7f0e640cdc50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0e54008fd0 tx=0x7f0e540015b0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 shutdown_connections 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0e5006c920 0x7f0e5006ede0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e640c98b0 0x7f0e640cdc50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0e640c9450 0x7f0e640cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 --2- 172.21.15.46:0/698207872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e4c004d40 0x7f0e6415fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 >> 172.21.15.46:0/698207872 conn(0x7f0e6408bf60 msgr2=0x7f0e640b9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 shutdown_connections 2024-04-15T15:16:57.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:57.905+0000 7f0e6bf59640 1 -- 172.21.15.46:0/698207872 wait complete. 2024-04-15T15:16:57.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:57.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:58.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.024+0000 7fc09e2c8640 1 Processor -- start 2024-04-15T15:16:58.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.024+0000 7fc09e2c8640 1 -- start start 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc09e2c8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc09815e830 0x7fc0980b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc09e2c8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc098067e30 con 0x7fc0980b53c0 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc09e2c8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0980b3c10 con 0x7fc09815ec10 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc09e2c8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc0980b3ef0 con 0x7fc09815e830 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc09c865640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc0980b53c0 0x7fc0980aeac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35556/0 (socket says 172.21.15.46:35556) 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc09c865640 1 -- 172.21.15.46:0/1681908194 learned_addr learned my addr 172.21.15.46:0/1681908194 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.025+0000 7fc097fff640 1 --2- 172.21.15.46:0/1681908194 >> v2:172.21.15.78:3300/0 conn(0x7fc09815e830 0x7fc0980b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308792571 0 0) 0x7fc098067e30 con 0x7fc0980b53c0 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc080003520 con 0x7fc0980b53c0 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69570114 0 0) 0x7fc0980b3c10 con 0x7fc09815ec10 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc098067e30 con 0x7fc09815ec10 2024-04-15T15:16:58.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1416290495 0 0) 0x7fc080003520 con 0x7fc0980b53c0 2024-04-15T15:16:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc097fff640 1 -- 172.21.15.46:0/1681908194 >> v1:172.21.15.78:6789/0 conn(0x7fc09815ec10 legacy=0x7fc098067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc097fff640 1 -- 172.21.15.46:0/1681908194 >> v1:172.21.15.46:6789/0 conn(0x7fc0980b53c0 legacy=0x7fc0980aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.026+0000 7fc097fff640 1 -- 172.21.15.46:0/1681908194 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0980b41d0 con 0x7fc09815e830 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.027+0000 7fc097fff640 1 --2- 172.21.15.46:0/1681908194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0980b4cb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc0880097a0 tx=0x7fc08802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d210ee9b6301b1d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.027+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088037070 con 0x7fc09815e830 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.027+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc088030510 con 0x7fc09815e830 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.027+0000 7fc096ffd640 1 -- 172.21.15.46:0/1681908194 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0900019f0 con 0x7fc09815e830 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1681908194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 msgr2=0x7fc0980b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1681908194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0980b4cb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc0880097a0 tx=0x7fc08802b790 comp rx=0 tx=0).stop 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1681908194 shutdown_connections 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1681908194 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0980b4cb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1681908194 >> 172.21.15.46:0/1681908194 conn(0x7fc098073360 msgr2=0x7fc09806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1681908194 shutdown_connections 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.028+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1681908194 wait complete. 2024-04-15T15:16:58.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.029+0000 7fc09e2c8640 1 Processor -- start 2024-04-15T15:16:58.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.029+0000 7fc09e2c8640 1 -- start start 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09e2c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 0x7fc098142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09e2c8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0981f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09e2c8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc098142c50 0x7fc0981f7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09e2c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc09815ff60 con 0x7fc0980b53c0 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09e2c8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc0980b4790 con 0x7fc09815e830 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09e2c8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc0981600e0 con 0x7fc098142c50 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 0x7fc098142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 0x7fc098142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47720/0 (socket says 172.21.15.46:47720) 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 -- 172.21.15.46:0/1112778595 learned_addr learned my addr 172.21.15.46:0/1112778595 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc09c865640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc098142c50 0x7fc0981f7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc0977fe640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0981f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 -- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc098142c50 msgr2=0x7fc0981f7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc098142c50 0x7fc0981f7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 -- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 msgr2=0x7fc0981f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0981f79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 -- 172.21.15.46:0/1112778595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0981f53d0 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc097fff640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 0x7fc098142350 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc0880097a0 tx=0x7fc0880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.030+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088042070 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.031+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc088004220 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.031+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0981f50f0 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.031+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc088037040 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.031+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc098155490 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.032+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0980689e0 con 0x7fc0980b53c0 2024-04-15T15:16:58.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.032+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc08803a040 con 0x7fc0980b53c0 2024-04-15T15:16:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.033+0000 7fc094ff9640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc08406c8f0 0x7fc08406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.033+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fc0880ad030 con 0x7fc0980b53c0 2024-04-15T15:16:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.033+0000 7fc0977fe640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc08406c8f0 0x7fc08406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.033+0000 7fc0977fe640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc08406c8f0 0x7fc08406edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0011e0 tx=0x7fc08c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.037+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc08807afd0 con 0x7fc0980b53c0 2024-04-15T15:16:58.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.218+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0980b5d50 con 0x7fc0980b53c0 2024-04-15T15:16:58.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.219+0000 7fc094ff9640 1 -- 172.21.15.46:0/1112778595 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7fc08807f2d0 con 0x7fc0980b53c0 2024-04-15T15:16:58.221 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:16:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc08406c8f0 msgr2=0x7fc08406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc08406c8f0 0x7fc08406edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0011e0 tx=0x7fc08c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 msgr2=0x7fc098142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 0x7fc098142350 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc0880097a0 tx=0x7fc0880046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 shutdown_connections 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc08406c8f0 0x7fc08406edb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc098142c50 0x7fc0981f7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc09815e830 0x7fc0981f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 --2- 172.21.15.46:0/1112778595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0980b53c0 0x7fc098142350 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 >> 172.21.15.46:0/1112778595 conn(0x7fc098073360 msgr2=0x7fc0980aeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 shutdown_connections 2024-04-15T15:16:58.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.221+0000 7fc09e2c8640 1 -- 172.21.15.46:0/1112778595 wait complete. 2024-04-15T15:16:58.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:16:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.337+0000 7f06cd614640 1 Processor -- start 2024-04-15T15:16:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.337+0000 7f06cd614640 1 -- start start 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.337+0000 7f06cd614640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c80d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06cd614640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f06c80d4a40 0x7f06c80d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06cd614640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c815eb70 con 0x7f06c815e790 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06cd614640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f06c80d3790 con 0x7f06c80d4a40 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06cd614640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f06c80d45b0 con 0x7f06c80d15e0 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f06c815e790 0x7f06c80b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35570/0 (socket says 172.21.15.46:35570) 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c6ffd640 1 -- 172.21.15.46:0/4241611141 learned_addr learned my addr 172.21.15.46:0/4241611141 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c77fe640 1 --2- 172.21.15.46:0/4241611141 >> v2:172.21.15.78:3300/0 conn(0x7f06c80d4a40 0x7f06c80d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06bffff640 1 --2- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c80d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c67fc640 1 -- 172.21.15.46:0/4241611141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457686849 0 0) 0x7f06c815eb70 con 0x7f06c815e790 2024-04-15T15:16:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c67fc640 1 -- 172.21.15.46:0/4241611141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06b4003520 con 0x7f06c815e790 2024-04-15T15:16:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c67fc640 1 -- 172.21.15.46:0/4241611141 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3139752844 0 0) 0x7f06b4003520 con 0x7f06c815e790 2024-04-15T15:16:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c77fe640 1 -- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 msgr2=0x7f06c80d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c77fe640 1 --2- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c80d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c77fe640 1 -- 172.21.15.46:0/4241611141 >> v1:172.21.15.46:6789/0 conn(0x7f06c815e790 legacy=0x7f06c80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.338+0000 7f06c77fe640 1 -- 172.21.15.46:0/4241611141 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c80d3c80 con 0x7f06c80d4a40 2024-04-15T15:16:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.339+0000 7f06c77fe640 1 --2- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c80d4a40 0x7f06c80d4eb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f06c80d3910 tx=0x7f06c002bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3b61e8f644bada2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.339+0000 7f06c67fc640 1 -- 172.21.15.46:0/4241611141 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c0039070 con 0x7f06c80d4a40 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.339+0000 7f06c67fc640 1 -- 172.21.15.46:0/4241611141 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c002c780 con 0x7f06c80d4a40 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.339+0000 7f06c67fc640 1 -- 172.21.15.46:0/4241611141 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8001690 con 0x7f06c80d4a40 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 -- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c80d4a40 msgr2=0x7f06c80d4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 --2- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c80d4a40 0x7f06c80d4eb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f06c80d3910 tx=0x7f06c002bc50 comp rx=0 tx=0).stop 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 -- 172.21.15.46:0/4241611141 shutdown_connections 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 --2- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c80d4a40 0x7f06c80d4eb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 --2- 172.21.15.46:0/4241611141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c80d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 -- 172.21.15.46:0/4241611141 >> 172.21.15.46:0/4241611141 conn(0x7f06c8067b40 msgr2=0x7f06c8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 -- 172.21.15.46:0/4241611141 shutdown_connections 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.340+0000 7f06cd614640 1 -- 172.21.15.46:0/4241611141 wait complete. 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.341+0000 7f06cd614640 1 Processor -- start 2024-04-15T15:16:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.341+0000 7f06cd614640 1 -- start start 2024-04-15T15:16:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06cd614640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c814dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06cd614640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 0x7f06c80c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06cd614640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c815e790 0x7f06c815fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06cd614640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f06c80d4240 con 0x7f06c80d4a40 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06cd614640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f06c80d53f0 con 0x7f06c815e790 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06cd614640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f06c80b7db0 con 0x7f06c80d15e0 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 0x7f06c80c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 0x7f06c80c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47724/0 (socket says 172.21.15.46:47724) 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 -- 172.21.15.46:0/362811494 learned_addr learned my addr 172.21.15.46:0/362811494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06c6ffd640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c814dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06c77fe640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c815e790 0x7f06c815fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 -- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 msgr2=0x7f06c814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c814dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 -- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c815e790 msgr2=0x7f06c815fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c815e790 0x7f06c815fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 -- 172.21.15.46:0/362811494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c80c8e10 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06c77fe640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c815e790 0x7f06c815fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.342+0000 7f06bffff640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 0x7f06c80c9eb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06b000e870 tx=0x7f06b000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.343+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b0017070 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.343+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b000c940 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.343+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c80c90f0 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.343+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b000cd30 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.343+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c80c96b0 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.344+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c80689e0 con 0x7f06c80d4a40 2024-04-15T15:16:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.344+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06b001a070 con 0x7f06c80d4a40 2024-04-15T15:16:58.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.345+0000 7f06bf7fe640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f06a806c8f0 0x7f06a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.345+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f06b008b7d0 con 0x7f06c80d4a40 2024-04-15T15:16:58.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.345+0000 7f06c6ffd640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f06a806c8f0 0x7f06a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.345+0000 7f06c6ffd640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f06a806c8f0 0x7f06a806edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f06b8001eb0 tx=0x7f06b8001d00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.349+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06b0059e50 con 0x7f06c80d4a40 2024-04-15T15:16:58.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.537+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f06c80b96d0 con 0x7f06c80d4a40 2024-04-15T15:16:58.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.538+0000 7f06bf7fe640 1 -- 172.21.15.46:0/362811494 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7f06b0059750 con 0x7f06c80d4a40 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:16:58.540 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f06a806c8f0 msgr2=0x7f06a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f06a806c8f0 0x7f06a806edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f06b8001eb0 tx=0x7f06b8001d00 comp rx=0 tx=0).stop 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 msgr2=0x7f06c80c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 0x7f06c80c9eb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f06b000e870 tx=0x7f06b000ed30 comp rx=0 tx=0).stop 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 shutdown_connections 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f06a806c8f0 0x7f06a806edb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06c815e790 0x7f06c815fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06c80d4a40 0x7f06c80c9eb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 --2- 172.21.15.46:0/362811494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f06c80d15e0 0x7f06c814dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 >> 172.21.15.46:0/362811494 conn(0x7f06c8067b40 msgr2=0x7f06c80b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 shutdown_connections 2024-04-15T15:16:58.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.540+0000 7f06cd614640 1 -- 172.21.15.46:0/362811494 wait complete. 2024-04-15T15:16:58.553 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:16:58.553 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:16:58.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.660+0000 7f5d167cd640 1 Processor -- start 2024-04-15T15:16:58.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.660+0000 7f5d167cd640 1 -- start start 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d167cd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5d1015e700 0x7f5d1009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d167cd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 0x7f5d1006d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d167cd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d100d09f0 con 0x7f5d1009b920 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d167cd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f5d100d1320 con 0x7f5d1015e700 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d167cd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d100d0cd0 con 0x7f5d100d17b0 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d14d6a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5d1009b920 0x7f5d100cdc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:35586/0 (socket says 172.21.15.46:35586) 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.661+0000 7f5d14d6a640 1 -- 172.21.15.46:0/1298081687 learned_addr learned my addr 172.21.15.46:0/1298081687 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0f7fe640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 0x7f5d1006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2501751670 0 0) 0x7f5d100d09f0 con 0x7f5d1009b920 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0ffff640 1 --2- 172.21.15.46:0/1298081687 >> v2:172.21.15.78:3300/0 conn(0x7f5d1015e700 0x7f5d1009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cf8003520 con 0x7f5d1009b920 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2438956441 0 0) 0x7f5cf8003520 con 0x7f5d1009b920 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 msgr2=0x7f5d1006d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 0x7f5d1006d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 >> v2:172.21.15.78:3300/0 conn(0x7f5d1015e700 msgr2=0x7f5d1009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 --2- 172.21.15.46:0/1298081687 >> v2:172.21.15.78:3300/0 conn(0x7f5d1015e700 0x7f5d1009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d100d0fb0 con 0x7f5d1009b920 2024-04-15T15:16:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f5d080019f0 con 0x7f5d1009b920 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 >> v1:172.21.15.46:6789/0 conn(0x7f5d1009b920 legacy=0x7f5d100cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 0x7f5cf80039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf8003f50 0x7f5cf8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5cf80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d100d1320 con 0x7f5cf8004d40 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5d100d0cd0 con 0x7f5cf8003f50 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf800b9c0 con 0x7f5d100d17b0 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d14d6a640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5cf80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0ffff640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 0x7f5cf80039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.662+0000 7f5d0f7fe640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf8003f50 0x7f5cf8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d14d6a640 1 -- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 msgr2=0x7f5cf80039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d14d6a640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d100d17b0 0x7f5cf80039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d14d6a640 1 -- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf8003f50 msgr2=0x7f5cf8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d14d6a640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf8003f50 0x7f5cf8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d14d6a640 1 -- 172.21.15.46:0/1298081687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf8003520 con 0x7f5cf8004d40 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d14d6a640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5cf80051c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800a890 tx=0x7f5cf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c062c52a29ab8e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800a480 con 0x7f5cf8004d40 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d08034630 con 0x7f5cf8004d40 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.663+0000 7f5d0effd640 1 -- 172.21.15.46:0/1298081687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08034aa0 con 0x7f5cf8004d40 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d167cd640 1 -- 172.21.15.46:0/1298081687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 msgr2=0x7f5cf80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d167cd640 1 --2- 172.21.15.46:0/1298081687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5cf80051c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800a890 tx=0x7f5cf8005830 comp rx=0 tx=0).stop 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d14d6a640 1 -- 172.21.15.46:0/1298081687 reap_dead start 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d167cd640 1 -- 172.21.15.46:0/1298081687 shutdown_connections 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d167cd640 1 -- 172.21.15.46:0/1298081687 >> 172.21.15.46:0/1298081687 conn(0x7f5d100b3ec0 msgr2=0x7f5d100b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d167cd640 1 -- 172.21.15.46:0/1298081687 shutdown_connections 2024-04-15T15:16:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.664+0000 7f5d167cd640 1 -- 172.21.15.46:0/1298081687 wait complete. 2024-04-15T15:16:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.665+0000 7f5d167cd640 1 Processor -- start 2024-04-15T15:16:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.665+0000 7f5d167cd640 1 -- start start 2024-04-15T15:16:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d167cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5d10160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d167cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d100b4ff0 0x7f5d1015f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d167cd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d1015f850 0x7f5d10071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d167cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5d100d0790 con 0x7f5cf8004d40 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d167cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5d100d0610 con 0x7f5d100b4ff0 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d167cd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1015eae0 con 0x7f5d1015f850 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5d10160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d0ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5d10160fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47732/0 (socket says 172.21.15.46:47732) 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.666+0000 7f5d0ffff640 1 -- 172.21.15.46:0/3881523753 learned_addr learned my addr 172.21.15.46:0/3881523753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0ffff640 1 -- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d1015f850 msgr2=0x7f5d10071120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0ffff640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d1015f850 0x7f5d10071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0ffff640 1 -- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d100b4ff0 msgr2=0x7f5d1015f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0ffff640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d100b4ff0 0x7f5d1015f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0ffff640 1 -- 172.21.15.46:0/3881523753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d10161800 con 0x7f5cf8004d40 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0ffff640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5d10160fd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5d00011aa0 tx=0x7f5d00011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0000fb00 con 0x7f5cf8004d40 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0000fca0 con 0x7f5cf8004d40 2024-04-15T15:16:58.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d10161ae0 con 0x7f5cf8004d40 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d00014660 con 0x7f5cf8004d40 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.667+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d100c7320 con 0x7f5cf8004d40 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.668+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d100690f0 con 0x7f5cf8004d40 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.669+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d00012040 con 0x7f5cf8004d40 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.669+0000 7f5d0cff9640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cfc06c8f0 0x7f5cfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.669+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f5d0008c0e0 con 0x7f5cf8004d40 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.669+0000 7f5d0f7fe640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cfc06c8f0 0x7f5cfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.670+0000 7f5d0f7fe640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cfc06c8f0 0x7f5cfc06edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d100c92c0 tx=0x7f5d04009320 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.674+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d00019030 con 0x7f5cf8004d40 2024-04-15T15:16:58.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:58.857+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5d100b8250 con 0x7f5cf8004d40 2024-04-15T15:16:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.032+0000 7f5d0cff9640 1 -- 172.21.15.46:0/3881523753 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7f5d0005a080 con 0x7f5cf8004d40 2024-04-15T15:16:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cfc06c8f0 msgr2=0x7f5cfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cfc06c8f0 0x7f5cfc06edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5d100c92c0 tx=0x7f5d04009320 comp rx=0 tx=0).stop 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 msgr2=0x7f5d10160fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5d10160fd0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5d00011aa0 tx=0x7f5d00011f60 comp rx=0 tx=0).stop 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 shutdown_connections 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cfc06c8f0 0x7f5cfc06edb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5d1015f850 0x7f5d10071120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5d100b4ff0 0x7f5d1015f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 --2- 172.21.15.46:0/3881523753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf8004d40 0x7f5d10160fd0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 >> 172.21.15.46:0/3881523753 conn(0x7f5d100b3ec0 msgr2=0x7f5d100b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 shutdown_connections 2024-04-15T15:16:59.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.035+0000 7f5d167cd640 1 -- 172.21.15.46:0/3881523753 wait complete. 2024-04-15T15:16:59.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:16:59.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.161+0000 7ff693d9d640 1 Processor -- start 2024-04-15T15:16:59.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.161+0000 7ff693d9d640 1 -- start start 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff693d9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 0x7ff68c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff693d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff68c0b7be0 0x7ff68c0d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff693d9d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff68c15eb70 con 0x7ff68c0d1b20 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff693d9d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7ff68c0d3790 con 0x7ff68c0b7be0 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff693d9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff68c0d45b0 con 0x7ff68c15e790 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff691b39640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 0x7ff68c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff691338640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff68c0b7be0 0x7ff68c0d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff691338640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff68c0b7be0 0x7ff68c0d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:60892/0 (socket says 172.21.15.46:60892) 2024-04-15T15:16:59.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff691b39640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 0x7ff68c0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:35458/0 (socket says 172.21.15.46:35458) 2024-04-15T15:16:59.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.162+0000 7ff691338640 1 -- 172.21.15.46:0/2388189208 learned_addr learned my addr 172.21.15.46:0/2388189208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.163+0000 7ff691338640 1 -- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 msgr2=0x7ff68c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.163+0000 7ff691338640 1 --2- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 0x7ff68c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.163+0000 7ff691338640 1 -- 172.21.15.46:0/2388189208 >> v1:172.21.15.78:6789/0 conn(0x7ff68c0d1b20 legacy=0x7ff68c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.163+0000 7ff691338640 1 -- 172.21.15.46:0/2388189208 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68c0d3c80 con 0x7ff68c0b7be0 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.163+0000 7ff691b39640 1 --2- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 0x7ff68c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff691338640 1 --2- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c0d15e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff680009870 tx=0x7ff68002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df968740a8c920c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff690b37640 1 -- 172.21.15.46:0/2388189208 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800317c0 con 0x7ff68c0b7be0 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff690b37640 1 -- 172.21.15.46:0/2388189208 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff680037070 con 0x7ff68c0b7be0 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff690b37640 1 -- 172.21.15.46:0/2388189208 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680031b80 con 0x7ff68c0b7be0 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 -- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 msgr2=0x7ff68c0d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 --2- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c0d15e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff680009870 tx=0x7ff68002b860 comp rx=0 tx=0).stop 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 -- 172.21.15.46:0/2388189208 shutdown_connections 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 --2- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c0d15e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 --2- 172.21.15.46:0/2388189208 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c15e790 0x7ff68c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 -- 172.21.15.46:0/2388189208 >> 172.21.15.46:0/2388189208 conn(0x7ff68c067b40 msgr2=0x7ff68c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 -- 172.21.15.46:0/2388189208 shutdown_connections 2024-04-15T15:16:59.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.164+0000 7ff693d9d640 1 -- 172.21.15.46:0/2388189208 wait complete. 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 Processor -- start 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 -- start start 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c161470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c0d1b20 0x7ff68c14b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 0x7ff68c159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c0d5150 con 0x7ff68c15e790 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c0d4240 con 0x7ff68c0b7be0 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff693d9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff68c0d52d0 con 0x7ff68c0d1b20 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff69233a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 0x7ff68c159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff69233a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 0x7ff68c159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47748/0 (socket says 172.21.15.46:47748) 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.166+0000 7ff69233a640 1 -- 172.21.15.46:0/1373232891 learned_addr learned my addr 172.21.15.46:0/1373232891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff691338640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c0d1b20 0x7ff68c14b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff691b39640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c161470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff69233a640 1 -- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c0d1b20 msgr2=0x7ff68c14b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff69233a640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c0d1b20 0x7ff68c14b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff69233a640 1 -- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 msgr2=0x7ff68c161470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff69233a640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c161470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff69233a640 1 -- 172.21.15.46:0/1373232891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68c15c520 con 0x7ff68c15e790 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff69233a640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 0x7ff68c159530 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff68400c680 tx=0x7ff68400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684003b90 con 0x7ff68c15e790 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff684003d30 con 0x7ff68c15e790 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68c15c800 con 0x7ff68c15e790 2024-04-15T15:16:59.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff684005310 con 0x7ff68c15e790 2024-04-15T15:16:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.167+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68c160c70 con 0x7ff68c15e790 2024-04-15T15:16:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.168+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68c0689e0 con 0x7ff68c15e790 2024-04-15T15:16:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.169+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff6840054b0 con 0x7ff68c15e790 2024-04-15T15:16:59.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.169+0000 7ff68a7fc640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff67806c8f0 0x7ff67806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.169+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7ff68408b850 con 0x7ff68c15e790 2024-04-15T15:16:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.169+0000 7ff691b39640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff67806c8f0 0x7ff67806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.170+0000 7ff691b39640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff67806c8f0 0x7ff67806edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009560 tx=0x7ff67c009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.174+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff684059600 con 0x7ff68c15e790 2024-04-15T15:16:59.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:16:59.357+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff68c161b80 con 0x7ff68c15e790 2024-04-15T15:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.035+0000 7ff68a7fc640 1 -- 172.21.15.46:0/1373232891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7ff68405d900 con 0x7ff68c15e790 2024-04-15T15:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-04-15T15:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff67806c8f0 msgr2=0x7ff67806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff67806c8f0 0x7ff67806edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009560 tx=0x7ff67c009190 comp rx=0 tx=0).stop 2024-04-15T15:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 msgr2=0x7ff68c159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 0x7ff68c159530 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff68400c680 tx=0x7ff68400cb40 comp rx=0 tx=0).stop 2024-04-15T15:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 shutdown_connections 2024-04-15T15:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff67806c8f0 0x7ff67806edb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff68c15e790 0x7ff68c159530 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff68c0d1b20 0x7ff68c14b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 --2- 172.21.15.46:0/1373232891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff68c0b7be0 0x7ff68c161470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 >> 172.21.15.46:0/1373232891 conn(0x7ff68c067b40 msgr2=0x7ff68c0d4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 shutdown_connections 2024-04-15T15:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.039+0000 7ff693d9d640 1 -- 172.21.15.46:0/1373232891 wait complete. 2024-04-15T15:17:00.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:17:00.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.179+0000 7f3d57150640 1 Processor -- start 2024-04-15T15:17:00.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.179+0000 7f3d57150640 1 -- start start 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.179+0000 7f3d57150640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3d500d3240 0x7f3d500d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.179+0000 7f3d57150640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3d500d1440 0x7f3d500b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d57150640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d500d2480 con 0x7f3d5015e790 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d57150640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3d500d2db0 con 0x7f3d500d3240 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d57150640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3d500d2760 con 0x7f3d500d1440 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3d500d3240 0x7f3d500d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3d500d3240 0x7f3d500d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48050/0 (socket says 172.21.15.46:48050) 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 -- 172.21.15.46:0/2269717611 learned_addr learned my addr 172.21.15.46:0/2269717611 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:00.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d556ed640 1 --2- 172.21.15.46:0/2269717611 >> v2:172.21.15.78:3300/0 conn(0x7f3d500d1440 0x7f3d500b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 -- 172.21.15.46:0/2269717611 >> v2:172.21.15.78:3300/0 conn(0x7f3d500d1440 msgr2=0x7f3d500b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 --2- 172.21.15.46:0/2269717611 >> v2:172.21.15.78:3300/0 conn(0x7f3d500d1440 0x7f3d500b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 -- 172.21.15.46:0/2269717611 >> v1:172.21.15.78:6789/0 conn(0x7f3d5015e790 legacy=0x7f3d500cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.180+0000 7f3d4ffff640 1 -- 172.21.15.46:0/2269717611 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d500d2a40 con 0x7f3d500d3240 2024-04-15T15:17:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d4ffff640 1 --2- 172.21.15.46:0/2269717611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d3240 0x7f3d500d0f00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c8120 tx=0x7f3d4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b77d0f1408f5ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d4f7fe640 1 -- 172.21.15.46:0/2269717611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4002ca80 con 0x7f3d500d3240 2024-04-15T15:17:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d4f7fe640 1 -- 172.21.15.46:0/2269717611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4002cc20 con 0x7f3d500d3240 2024-04-15T15:17:00.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d4f7fe640 1 -- 172.21.15.46:0/2269717611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d400343f0 con 0x7f3d500d3240 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d57150640 1 -- 172.21.15.46:0/2269717611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d3240 msgr2=0x7f3d500d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d57150640 1 --2- 172.21.15.46:0/2269717611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d3240 0x7f3d500d0f00 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3d500c8120 tx=0x7f3d4002bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d57150640 1 -- 172.21.15.46:0/2269717611 shutdown_connections 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d57150640 1 --2- 172.21.15.46:0/2269717611 >> v2:172.21.15.78:3300/0 conn(0x7f3d500d1440 0x7f3d500b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d57150640 1 --2- 172.21.15.46:0/2269717611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d3240 0x7f3d500d0f00 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.181+0000 7f3d57150640 1 -- 172.21.15.46:0/2269717611 >> 172.21.15.46:0/2269717611 conn(0x7f3d5008bf60 msgr2=0x7f3d500d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.182+0000 7f3d57150640 1 -- 172.21.15.46:0/2269717611 shutdown_connections 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.182+0000 7f3d57150640 1 -- 172.21.15.46:0/2269717611 wait complete. 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.183+0000 7f3d57150640 1 Processor -- start 2024-04-15T15:17:00.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.183+0000 7f3d57150640 1 -- start start 2024-04-15T15:17:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 0x7f3d5014ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3d500d3240 0x7f3d500ca240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d5015e790 0x7f3d500b9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3d500d5180 con 0x7f3d500d1440 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3d500d0430 con 0x7f3d5015e790 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3d500d5300 con 0x7f3d500d3240 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 0x7f3d5014ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 0x7f3d5014ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48052/0 (socket says 172.21.15.46:48052) 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 -- 172.21.15.46:0/2663277976 learned_addr learned my addr 172.21.15.46:0/2663277976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d556ed640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d5015e790 0x7f3d500b9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d4ffff640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3d500d3240 0x7f3d500ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 -- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3d500d3240 msgr2=0x7f3d500ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3d500d3240 0x7f3d500ca240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 -- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d5015e790 msgr2=0x7f3d500b9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d5015e790 0x7f3d500b9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 -- 172.21.15.46:0/2663277976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d50159060 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d54eec640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 0x7f3d5014ba40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b3f0 tx=0x7f3d4800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48015020 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d48011830 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5015a570 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48011c20 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.184+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5015ab30 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.185+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d500689e0 con 0x7f3d500d1440 2024-04-15T15:17:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.186+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d48018020 con 0x7f3d500d1440 2024-04-15T15:17:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.186+0000 7f3d4d7fa640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3d3006c8f0 0x7f3d3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.187+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7f3d4808b870 con 0x7f3d500d1440 2024-04-15T15:17:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.187+0000 7f3d4ffff640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3d3006c8f0 0x7f3d3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.187+0000 7f3d4ffff640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3d3006c8f0 0x7f3d3006edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3d40035000 tx=0x7f3d40030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.191+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d4801f030 con 0x7f3d500d1440 2024-04-15T15:17:00.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:00.417+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3d500d1ce0 con 0x7f3d500d1440 2024-04-15T15:17:01.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.054+0000 7f3d4d7fa640 1 -- 172.21.15.46:0/2663277976 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7f3d48059480 con 0x7f3d500d1440 2024-04-15T15:17:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 5 and data pool 6 2024-04-15T15:17:01.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.070+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3d3006c8f0 msgr2=0x7f3d3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.070+0000 7f3d57150640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3d3006c8f0 0x7f3d3006edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3d40035000 tx=0x7f3d40030000 comp rx=0 tx=0).stop 2024-04-15T15:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.070+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 msgr2=0x7f3d5014ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.070+0000 7f3d57150640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 0x7f3d5014ba40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b3f0 tx=0x7f3d4800b8b0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 shutdown_connections 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3d3006c8f0 0x7f3d3006edb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d5015e790 0x7f3d500b9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3d500d3240 0x7f3d500ca240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 --2- 172.21.15.46:0/2663277976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d500d1440 0x7f3d5014ba40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 >> 172.21.15.46:0/2663277976 conn(0x7f3d5008bf60 msgr2=0x7f3d500cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.071+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 shutdown_connections 2024-04-15T15:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.072+0000 7f3d57150640 1 -- 172.21.15.46:0/2663277976 wait complete. 2024-04-15T15:17:01.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:01.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 Processor -- start 2024-04-15T15:17:01.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 -- start start 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e00d15e0 0x7f79e00d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79e00d4a40 0x7f79e00d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79e015eb70 con 0x7f79e015e790 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f79e00d3790 con 0x7f79e00d4a40 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.200+0000 7f79e501b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f79e00d45b0 con 0x7f79e00d15e0 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79df59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f79e00d4a40 0x7f79e00d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79ded9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f79e015e790 0x7f79e00b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52202/0 (socket says 172.21.15.46:52202) 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79ded9d640 1 -- 172.21.15.46:0/3250065418 learned_addr learned my addr 172.21.15.46:0/3250065418 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79de59c640 1 --2- 172.21.15.46:0/3250065418 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e00d15e0 0x7f79e00d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79df59e640 1 -- 172.21.15.46:0/3250065418 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e00d15e0 msgr2=0x7f79e00d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79df59e640 1 --2- 172.21.15.46:0/3250065418 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e00d15e0 0x7f79e00d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79df59e640 1 -- 172.21.15.46:0/3250065418 >> v1:172.21.15.46:6789/0 conn(0x7f79e015e790 legacy=0x7f79e00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79df59e640 1 -- 172.21.15.46:0/3250065418 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00d3c80 con 0x7f79e00d4a40 2024-04-15T15:17:01.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.201+0000 7f79ddd9b640 1 -- 172.21.15.46:0/3250065418 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075013910 0 0) 0x7f79e015eb70 con 0x7f79e015e790 2024-04-15T15:17:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79df59e640 1 --2- 172.21.15.46:0/3250065418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d4a40 0x7f79e00d4eb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f79e00d3910 tx=0x7f79d802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a529367bd62aad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79ddd9b640 1 -- 172.21.15.46:0/3250065418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8039070 con 0x7f79e00d4a40 2024-04-15T15:17:01.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79ddd9b640 1 -- 172.21.15.46:0/3250065418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d80316f0 con 0x7f79e00d4a40 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79ddd9b640 1 -- 172.21.15.46:0/3250065418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d8031ac0 con 0x7f79e00d4a40 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 -- 172.21.15.46:0/3250065418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d4a40 msgr2=0x7f79e00d4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 --2- 172.21.15.46:0/3250065418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d4a40 0x7f79e00d4eb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f79e00d3910 tx=0x7f79d802b790 comp rx=0 tx=0).stop 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 -- 172.21.15.46:0/3250065418 shutdown_connections 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 --2- 172.21.15.46:0/3250065418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d4a40 0x7f79e00d4eb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 --2- 172.21.15.46:0/3250065418 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e00d15e0 0x7f79e00d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 -- 172.21.15.46:0/3250065418 >> 172.21.15.46:0/3250065418 conn(0x7f79e0067b40 msgr2=0x7f79e0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 -- 172.21.15.46:0/3250065418 shutdown_connections 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.202+0000 7f79e501b640 1 -- 172.21.15.46:0/3250065418 wait complete. 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.204+0000 7f79e501b640 1 Processor -- start 2024-04-15T15:17:01.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.204+0000 7f79e501b640 1 -- start start 2024-04-15T15:17:01.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 0x7f79e00c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e00d4a40 0x7f79e00cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e015e790 0x7f79e00b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f79e00d4240 con 0x7f79e00d15e0 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f79e00d53f0 con 0x7f79e00d4a40 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f79e00b7db0 con 0x7f79e015e790 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 0x7f79e00c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 0x7f79e00c8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48074/0 (socket says 172.21.15.46:48074) 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 -- 172.21.15.46:0/2219613531 learned_addr learned my addr 172.21.15.46:0/2219613531 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79de59c640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e00d4a40 0x7f79e00cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79df59e640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e015e790 0x7f79e00b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 -- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e015e790 msgr2=0x7f79e00b99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e015e790 0x7f79e00b99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 -- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e00d4a40 msgr2=0x7f79e00cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e00d4a40 0x7f79e00cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 -- 172.21.15.46:0/2219613531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e00cd6f0 con 0x7f79e00d15e0 2024-04-15T15:17:01.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79de59c640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e00d4a40 0x7f79e00cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79df59e640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e015e790 0x7f79e00b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79ded9d640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 0x7f79e00c8ee0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79d000cc10 tx=0x7f79d000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0017070 con 0x7f79e00d15e0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d000ce20 con 0x7f79e00d15e0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0012640 con 0x7f79e00d15e0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.205+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79e00c8120 con 0x7f79e00d15e0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.206+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79e00c86e0 con 0x7f79e00d15e0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.206+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79e00689e0 con 0x7f79e00d15e0 2024-04-15T15:17:01.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.207+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79d001a070 con 0x7f79e00d15e0 2024-04-15T15:17:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.207+0000 7f79d77fe640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79c006c8f0 0x7f79c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.207+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f79d008bb10 con 0x7f79e00d15e0 2024-04-15T15:17:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.208+0000 7f79de59c640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79c006c8f0 0x7f79c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.208+0000 7f79de59c640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79c006c8f0 0x7f79c006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79e00cf740 tx=0x7f79c8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.212+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79d0059dc0 con 0x7f79e00d15e0 2024-04-15T15:17:01.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.396+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79e00b6b00 con 0x7f79e00d15e0 2024-04-15T15:17:01.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.397+0000 7f79d77fe640 1 -- 172.21.15.46:0/2219613531 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16907 (secure 0 0 0) 0x7f79d00596c0 con 0x7f79e00d15e0 2024-04-15T15:17:01.399 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:01.399 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:00.418503+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79c006c8f0 msgr2=0x7f79c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79c006c8f0 0x7f79c006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79e00cf740 tx=0x7f79c8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 msgr2=0x7f79e00c8ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 0x7f79e00c8ee0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f79d000cc10 tx=0x7f79d000a430 comp rx=0 tx=0).stop 2024-04-15T15:17:01.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 shutdown_connections 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f79c006c8f0 0x7f79c006edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f79e015e790 0x7f79e00b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79e00d4a40 0x7f79e00cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 --2- 172.21.15.46:0/2219613531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79e00d15e0 0x7f79e00c8ee0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 >> 172.21.15.46:0/2219613531 conn(0x7f79e0067b40 msgr2=0x7f79e00b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 shutdown_connections 2024-04-15T15:17:01.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.399+0000 7f79e501b640 1 -- 172.21.15.46:0/2219613531 wait complete. 2024-04-15T15:17:01.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.515+0000 7f0b52fa1640 1 Processor -- start 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.516+0000 7f0b52fa1640 1 -- start start 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.516+0000 7f0b52fa1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b4c09aa60 0x7f0b4c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.516+0000 7f0b52fa1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4c0af2c0 con 0x7f0b4c06bf20 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.516+0000 7f0b52fa1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4c0af5a0 con 0x7f0b4c15e8f0 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.516+0000 7f0b52fa1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4c0b0080 con 0x7f0b4c09aa60 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.516+0000 7f0b527a0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0b4c06bf20 0x7f0b4c0b0770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52206/0 (socket says 172.21.15.46:52206) 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b527a0640 1 -- 172.21.15.46:0/3595465997 learned_addr learned my addr 172.21.15.46:0/3595465997 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b5179e640 1 --2- 172.21.15.46:0/3595465997 >> v2:172.21.15.78:3300/0 conn(0x7f0b4c09aa60 0x7f0b4c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b50f9d640 1 -- 172.21.15.46:0/3595465997 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399910502 0 0) 0x7f0b4c0af2c0 con 0x7f0b4c06bf20 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b50f9d640 1 -- 172.21.15.46:0/3595465997 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b34003520 con 0x7f0b4c06bf20 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b50f9d640 1 -- 172.21.15.46:0/3595465997 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1638696661 0 0) 0x7f0b34003520 con 0x7f0b4c06bf20 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b5179e640 1 -- 172.21.15.46:0/3595465997 >> v1:172.21.15.78:6789/0 conn(0x7f0b4c15e8f0 legacy=0x7f0b4c0b5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b5179e640 1 -- 172.21.15.46:0/3595465997 >> v1:172.21.15.46:6789/0 conn(0x7f0b4c06bf20 legacy=0x7f0b4c0b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.517+0000 7f0b5179e640 1 -- 172.21.15.46:0/3595465997 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c0af880 con 0x7f0b4c09aa60 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.518+0000 7f0b5179e640 1 --2- 172.21.15.46:0/3595465997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c06b9e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0b400097a0 tx=0x7f0b4002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f29c92f6409e509 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.518+0000 7f0b50f9d640 1 -- 172.21.15.46:0/3595465997 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40038070 con 0x7f0b4c09aa60 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.518+0000 7f0b50f9d640 1 -- 172.21.15.46:0/3595465997 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b40031500 con 0x7f0b4c09aa60 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.518+0000 7f0b50f9d640 1 -- 172.21.15.46:0/3595465997 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0019f0 con 0x7f0b4c09aa60 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3595465997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c09aa60 msgr2=0x7f0b4c06b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3595465997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c06b9e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0b400097a0 tx=0x7f0b4002b820 comp rx=0 tx=0).stop 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3595465997 shutdown_connections 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3595465997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c06b9e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3595465997 >> 172.21.15.46:0/3595465997 conn(0x7f0b4c073060 msgr2=0x7f0b4c0b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3595465997 shutdown_connections 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.519+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3595465997 wait complete. 2024-04-15T15:17:01.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.520+0000 7f0b52fa1640 1 Processor -- start 2024-04-15T15:17:01.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.520+0000 7f0b52fa1640 1 -- start start 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c06bf20 0x7f0b4c09ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c15ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b4c15e8f0 0x7f0b4c158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c0aefe0 con 0x7f0b4c09aa60 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c06c370 con 0x7f0b4c06bf20 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b4c0b0e80 con 0x7f0b4c15e8f0 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c15ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c15ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48084/0 (socket says 172.21.15.46:48084) 2024-04-15T15:17:01.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 -- 172.21.15.46:0/3421501361 learned_addr learned my addr 172.21.15.46:0/3421501361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b51f9f640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c06bf20 0x7f0b4c09ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b527a0640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b4c15e8f0 0x7f0b4c158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 -- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b4c15e8f0 msgr2=0x7f0b4c158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b4c15e8f0 0x7f0b4c158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 -- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c06bf20 msgr2=0x7f0b4c09ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c06bf20 0x7f0b4c09ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 -- 172.21.15.46:0/3421501361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c09f100 con 0x7f0b4c09aa60 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b5179e640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c15ef30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0b4002b860 tx=0x7f0b400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40043070 con 0x7f0b4c09aa60 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b4003f870 con 0x7f0b4c09aa60 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4c15f9f0 con 0x7f0b4c09aa60 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.521+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40038040 con 0x7f0b4c09aa60 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.522+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4c15ffb0 con 0x7f0b4c09aa60 2024-04-15T15:17:01.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.522+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4c04fd50 con 0x7f0b4c09aa60 2024-04-15T15:17:01.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.523+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b40046070 con 0x7f0b4c09aa60 2024-04-15T15:17:01.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.523+0000 7f0b4a7fc640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b3806c8f0 0x7f0b3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.524+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f0b400ad780 con 0x7f0b4c09aa60 2024-04-15T15:17:01.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.524+0000 7f0b51f9f640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b3806c8f0 0x7f0b3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.524+0000 7f0b51f9f640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b3806c8f0 0x7f0b3806edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c161ef0 tx=0x7f0b44009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.528+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b40030070 con 0x7f0b4c09aa60 2024-04-15T15:17:01.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.718+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0b4c0b07b0 con 0x7f0b4c09aa60 2024-04-15T15:17:01.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.720+0000 7f0b4a7fc640 1 -- 172.21.15.46:0/3421501361 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16907 (secure 0 0 0) 0x7f0b4007b460 con 0x7f0b4c09aa60 2024-04-15T15:17:01.722 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:01.722 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":38,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:00.418503+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b3806c8f0 msgr2=0x7f0b3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b3806c8f0 0x7f0b3806edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c161ef0 tx=0x7f0b44009040 comp rx=0 tx=0).stop 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 msgr2=0x7f0b4c15ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c15ef30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0b4002b860 tx=0x7f0b400046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 shutdown_connections 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b3806c8f0 0x7f0b3806edb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b4c15e8f0 0x7f0b4c158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4c09aa60 0x7f0b4c15ef30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 --2- 172.21.15.46:0/3421501361 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b4c06bf20 0x7f0b4c09ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 >> 172.21.15.46:0/3421501361 conn(0x7f0b4c073060 msgr2=0x7f0b4c070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 shutdown_connections 2024-04-15T15:17:01.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.722+0000 7f0b52fa1640 1 -- 172.21.15.46:0/3421501361 wait complete. 2024-04-15T15:17:01.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:17:01.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.841+0000 7f085251c640 1 Processor -- start 2024-04-15T15:17:01.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.841+0000 7f085251c640 1 -- start start 2024-04-15T15:17:01.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f085251c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f084c0d3240 0x7f084c0d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f085251c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f084c0d1440 0x7f084c0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f085251c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084c0d2480 con 0x7f084c15e790 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f085251c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f084c0d2db0 con 0x7f084c0d3240 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f085251c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f084c0d2760 con 0x7f084c0d1440 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f0843fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f084c0d3240 0x7f084c0d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f084bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f084c15e790 0x7f084c0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52208/0 (socket says 172.21.15.46:52208) 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f084bfff640 1 -- 172.21.15.46:0/854857008 learned_addr learned my addr 172.21.15.46:0/854857008 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f0850ab9640 1 --2- 172.21.15.46:0/854857008 >> v2:172.21.15.78:3300/0 conn(0x7f084c0d1440 0x7f084c0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f0843fff640 1 -- 172.21.15.46:0/854857008 >> v2:172.21.15.78:3300/0 conn(0x7f084c0d1440 msgr2=0x7f084c0b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.842+0000 7f0843fff640 1 --2- 172.21.15.46:0/854857008 >> v2:172.21.15.78:3300/0 conn(0x7f084c0d1440 0x7f084c0b74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.843+0000 7f0843fff640 1 -- 172.21.15.46:0/854857008 >> v1:172.21.15.46:6789/0 conn(0x7f084c15e790 legacy=0x7f084c0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.843+0000 7f0843fff640 1 -- 172.21.15.46:0/854857008 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c0d2a40 con 0x7f084c0d3240 2024-04-15T15:17:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.843+0000 7f0850ab9640 1 --2- 172.21.15.46:0/854857008 >> v2:172.21.15.78:3300/0 conn(0x7f084c0d1440 0x7f084c0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.843+0000 7f0843fff640 1 --2- 172.21.15.46:0/854857008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c0d3240 0x7f084c0d0f00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f084c0c8120 tx=0x7f083402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ec5b53a910758d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.843+0000 7f084b7fe640 1 -- 172.21.15.46:0/854857008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834035020 con 0x7f084c0d3240 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f084b7fe640 1 -- 172.21.15.46:0/854857008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0834031830 con 0x7f084c0d3240 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f084b7fe640 1 -- 172.21.15.46:0/854857008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834031c00 con 0x7f084c0d3240 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 -- 172.21.15.46:0/854857008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c0d3240 msgr2=0x7f084c0d0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 --2- 172.21.15.46:0/854857008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c0d3240 0x7f084c0d0f00 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f084c0c8120 tx=0x7f083402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 -- 172.21.15.46:0/854857008 shutdown_connections 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 --2- 172.21.15.46:0/854857008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c0d3240 0x7f084c0d0f00 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 --2- 172.21.15.46:0/854857008 >> v2:172.21.15.78:3300/0 conn(0x7f084c0d1440 0x7f084c0b74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 -- 172.21.15.46:0/854857008 >> 172.21.15.46:0/854857008 conn(0x7f084c08bf60 msgr2=0x7f084c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 -- 172.21.15.46:0/854857008 shutdown_connections 2024-04-15T15:17:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.844+0000 7f085251c640 1 -- 172.21.15.46:0/854857008 wait complete. 2024-04-15T15:17:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.845+0000 7f085251c640 1 Processor -- start 2024-04-15T15:17:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.845+0000 7f085251c640 1 -- start start 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f085251c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0d1440 0x7f084c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f085251c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f084c0d3240 0x7f084c0b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f085251c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 0x7f084c0b8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f085251c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f084c0d5180 con 0x7f084c15e790 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f085251c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f084c0d0430 con 0x7f084c0d1440 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f085251c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f084c0d5300 con 0x7f084c0d3240 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f0850ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 0x7f084c0b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f0850ab9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 0x7f084c0b8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48108/0 (socket says 172.21.15.46:48108) 2024-04-15T15:17:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f0850ab9640 1 -- 172.21.15.46:0/3112372650 learned_addr learned my addr 172.21.15.46:0/3112372650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f084bfff640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0d1440 0x7f084c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.846+0000 7f0843fff640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f084c0d3240 0x7f084c0b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0850ab9640 1 -- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f084c0d3240 msgr2=0x7f084c0b82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0850ab9640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f084c0d3240 0x7f084c0b82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0850ab9640 1 -- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0d1440 msgr2=0x7f084c152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0850ab9640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0d1440 0x7f084c152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0850ab9640 1 -- 172.21.15.46:0/3112372650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c1f8a40 con 0x7f084c15e790 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0843fff640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f084c0d3240 0x7f084c0b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f0850ab9640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 0x7f084c0b8800 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f084400cf20 tx=0x7f084400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084401a070 con 0x7f084c15e790 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f084400d840 con 0x7f084c15e790 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c0ca6c0 con 0x7f084c15e790 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084400dc30 con 0x7f084c15e790 2024-04-15T15:17:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.847+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c0cac80 con 0x7f084c15e790 2024-04-15T15:17:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.848+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c0b9120 con 0x7f084c15e790 2024-04-15T15:17:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.849+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f084401d070 con 0x7f084c15e790 2024-04-15T15:17:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.849+0000 7f08497fa640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f082c06c8f0 0x7f082c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.849+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f084408c4f0 con 0x7f084c15e790 2024-04-15T15:17:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.849+0000 7f084bfff640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f082c06c8f0 0x7f082c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.850+0000 7f084bfff640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f082c06c8f0 0x7f082c06edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f084c075340 tx=0x7f083c007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:01.853+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f084c0b9120 con 0x7f084c15e790 2024-04-15T15:17:02.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:02.082+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f084c073c40 con 0x7f084c15e790 2024-04-15T15:17:03.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.063+0000 7f08497fa640 1 -- 172.21.15.46:0/3112372650 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f084c073c40 con 0x7f084c15e790 2024-04-15T15:17:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.066+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f082c06c8f0 msgr2=0x7f082c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.066+0000 7f085251c640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f082c06c8f0 0x7f082c06edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f084c075340 tx=0x7f083c007e80 comp rx=0 tx=0).stop 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.066+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 msgr2=0x7f084c0b8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.066+0000 7f085251c640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 0x7f084c0b8800 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f084400cf20 tx=0x7f084400a7d0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 shutdown_connections 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f082c06c8f0 0x7f082c06edb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c15e790 0x7f084c0b8800 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f084c0d3240 0x7f084c0b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 --2- 172.21.15.46:0/3112372650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0d1440 0x7f084c152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 >> 172.21.15.46:0/3112372650 conn(0x7f084c08bf60 msgr2=0x7f084c0cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 shutdown_connections 2024-04-15T15:17:03.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.067+0000 7f085251c640 1 -- 172.21.15.46:0/3112372650 wait complete. 2024-04-15T15:17:03.082 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-15T15:17:03.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:17:03.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.194+0000 7f73fc088640 1 Processor -- start 2024-04-15T15:17:03.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.195+0000 7f73fc088640 1 -- start start 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.195+0000 7f73fc088640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f40cfa90 0x7f73f40afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.195+0000 7f73fc088640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f40cff00 con 0x7f73f40cf600 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.195+0000 7f73fc088640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f40ae440 con 0x7f73f415ed00 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.195+0000 7f73fc088640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f73f4071030 con 0x7f73f40cfa90 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f9623640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73f40cf600 0x7f73f40af4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52222/0 (socket says 172.21.15.46:52222) 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f9623640 1 -- 172.21.15.46:0/2249163859 learned_addr learned my addr 172.21.15.46:0/2249163859 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73fa625640 1 --2- 172.21.15.46:0/2249163859 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f40cfa90 0x7f73f40afbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:03.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3987895637 0 0) 0x7f73f40cff00 con 0x7f73f40cf600 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73dc003520 con 0x7f73f40cf600 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021515179 0 0) 0x7f73f40ae440 con 0x7f73f415ed00 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f40cff00 con 0x7f73f415ed00 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1154862511 0 0) 0x7f73dc003520 con 0x7f73f40cf600 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73fa625640 1 -- 172.21.15.46:0/2249163859 >> v1:172.21.15.78:6789/0 conn(0x7f73f415ed00 legacy=0x7f73f4070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73fa625640 1 -- 172.21.15.46:0/2249163859 >> v1:172.21.15.46:6789/0 conn(0x7f73f40cf600 legacy=0x7f73f40af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.196+0000 7f73fa625640 1 -- 172.21.15.46:0/2249163859 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f40ae720 con 0x7f73f40cfa90 2024-04-15T15:17:03.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.197+0000 7f73fa625640 1 --2- 172.21.15.46:0/2249163859 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f40cfa90 0x7f73f40afbf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0097d0 tx=0x7f73ec02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39a72d06b65e5776 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:03.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.197+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec037070 con 0x7f73f40cfa90 2024-04-15T15:17:03.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.197+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ec030510 con 0x7f73f40cfa90 2024-04-15T15:17:03.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.197+0000 7f73f8e22640 1 -- 172.21.15.46:0/2249163859 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8001670 con 0x7f73f40cfa90 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 -- 172.21.15.46:0/2249163859 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f40cfa90 msgr2=0x7f73f40afbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 --2- 172.21.15.46:0/2249163859 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f40cfa90 0x7f73f40afbf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0097d0 tx=0x7f73ec02b800 comp rx=0 tx=0).stop 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 -- 172.21.15.46:0/2249163859 shutdown_connections 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 --2- 172.21.15.46:0/2249163859 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f40cfa90 0x7f73f40afbf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 -- 172.21.15.46:0/2249163859 >> 172.21.15.46:0/2249163859 conn(0x7f73f409a9d0 msgr2=0x7f73f40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 -- 172.21.15.46:0/2249163859 shutdown_connections 2024-04-15T15:17:03.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.198+0000 7f73fc088640 1 -- 172.21.15.46:0/2249163859 wait complete. 2024-04-15T15:17:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.199+0000 7f73fc088640 1 Processor -- start 2024-04-15T15:17:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.199+0000 7f73fc088640 1 -- start start 2024-04-15T15:17:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fc088640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 0x7f73f41f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fc088640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f40cfa90 0x7f73f41f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fc088640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f415ed00 0x7f73f40c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fc088640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f73f40aefc0 con 0x7f73f40cf600 2024-04-15T15:17:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fc088640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f73f40aece0 con 0x7f73f40cfa90 2024-04-15T15:17:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fc088640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f73f40701a0 con 0x7f73f415ed00 2024-04-15T15:17:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 0x7f73f41f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 0x7f73f41f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48112/0 (socket says 172.21.15.46:48112) 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 -- 172.21.15.46:0/2179780632 learned_addr learned my addr 172.21.15.46:0/2179780632 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9623640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f40cfa90 0x7f73f41f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73fa625640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f415ed00 0x7f73f40c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 -- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f415ed00 msgr2=0x7f73f40c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f415ed00 0x7f73f40c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 -- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f40cfa90 msgr2=0x7f73f41f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f40cfa90 0x7f73f41f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 -- 172.21.15.46:0/2179780632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f40c9b50 con 0x7f73f40cf600 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9623640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f40cfa90 0x7f73f41f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f9e24640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 0x7f73f41f5820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f73e400c800 tx=0x7f73e400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.200+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4003b70 con 0x7f73f40cf600 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.201+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e4003d10 con 0x7f73f40cf600 2024-04-15T15:17:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.201+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f40c9e30 con 0x7f73f40cf600 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.201+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e400aa90 con 0x7f73f40cf600 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.201+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f40ca3f0 con 0x7f73f40cf600 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.201+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f40690f0 con 0x7f73f40cf600 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.202+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f73e400ac30 con 0x7f73f40cf600 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.202+0000 7f73f27fc640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f73e006c8f0 0x7f73e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.203+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f73e408bd10 con 0x7f73f40cf600 2024-04-15T15:17:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.203+0000 7f73f9623640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f73e006c8f0 0x7f73e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:03.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.203+0000 7f73f9623640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f73e006c8f0 0x7f73e006edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f73e80017e0 tx=0x7f73e8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:03.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.207+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f73e405a040 con 0x7f73f40cf600 2024-04-15T15:17:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:03.435+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f73f40b6520 con 0x7f73f40cf600 2024-04-15T15:17:04.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.069+0000 7f73f27fc640 1 -- 172.21.15.46:0/2179780632 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7f73e4059940 con 0x7f73f40cf600 2024-04-15T15:17:04.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f73e006c8f0 msgr2=0x7f73e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f73e006c8f0 0x7f73e006edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f73e80017e0 tx=0x7f73e8009040 comp rx=0 tx=0).stop 2024-04-15T15:17:04.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 msgr2=0x7f73f41f5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 0x7f73f41f5820 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f73e400c800 tx=0x7f73e400ccc0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 shutdown_connections 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f73e006c8f0 0x7f73e006edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f73f415ed00 0x7f73f40c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73f40cfa90 0x7f73f41f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 --2- 172.21.15.46:0/2179780632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73f40cf600 0x7f73f41f5820 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 >> 172.21.15.46:0/2179780632 conn(0x7f73f409a9d0 msgr2=0x7f73f40b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 shutdown_connections 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.072+0000 7f73fc088640 1 -- 172.21.15.46:0/2179780632 wait complete. 2024-04-15T15:17:04.088 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:17:04.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:04.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.197+0000 7f34508ba640 1 Processor -- start 2024-04-15T15:17:04.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.197+0000 7f34508ba640 1 -- start start 2024-04-15T15:17:04.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f34508ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f344c0d15e0 0x7f344c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f34508ba640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f344c0d4a40 0x7f344c0d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f34508ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f344c15eb70 con 0x7f344c15e790 2024-04-15T15:17:04.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f34508ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f344c0d3790 con 0x7f344c0d15e0 2024-04-15T15:17:04.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f34508ba640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f344c0d45b0 con 0x7f344c0d4a40 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f344c0d15e0 0x7f344c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f344c0d15e0 0x7f344c0d1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48120/0 (socket says 172.21.15.46:48120) 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 -- 172.21.15.46:0/2741332244 learned_addr learned my addr 172.21.15.46:0/2741332244 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 -- 172.21.15.46:0/2741332244 >> v2:172.21.15.78:3300/0 conn(0x7f344c0d4a40 msgr2=0x7f344c0d4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 --2- 172.21.15.46:0/2741332244 >> v2:172.21.15.78:3300/0 conn(0x7f344c0d4a40 0x7f344c0d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 -- 172.21.15.46:0/2741332244 >> v1:172.21.15.78:6789/0 conn(0x7f344c15e790 legacy=0x7f344c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.198+0000 7f3449d9b640 1 -- 172.21.15.46:0/2741332244 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c0d3c80 con 0x7f344c0d15e0 2024-04-15T15:17:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.199+0000 7f3449d9b640 1 --2- 172.21.15.46:0/2741332244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c0d15e0 0x7f344c0d1a90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f344c0d5690 tx=0x7f343402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5910713aa8b34e03 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.199+0000 7f344959a640 1 -- 172.21.15.46:0/2741332244 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434038070 con 0x7f344c0d15e0 2024-04-15T15:17:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.199+0000 7f344959a640 1 -- 172.21.15.46:0/2741332244 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f343402f050 con 0x7f344c0d15e0 2024-04-15T15:17:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.199+0000 7f344959a640 1 -- 172.21.15.46:0/2741332244 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434031970 con 0x7f344c0d15e0 2024-04-15T15:17:04.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 -- 172.21.15.46:0/2741332244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c0d15e0 msgr2=0x7f344c0d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 --2- 172.21.15.46:0/2741332244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c0d15e0 0x7f344c0d1a90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f344c0d5690 tx=0x7f343402b7f0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 -- 172.21.15.46:0/2741332244 shutdown_connections 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 --2- 172.21.15.46:0/2741332244 >> v2:172.21.15.78:3300/0 conn(0x7f344c0d4a40 0x7f344c0d4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 --2- 172.21.15.46:0/2741332244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c0d15e0 0x7f344c0d1a90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 -- 172.21.15.46:0/2741332244 >> 172.21.15.46:0/2741332244 conn(0x7f344c067b40 msgr2=0x7f344c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 -- 172.21.15.46:0/2741332244 shutdown_connections 2024-04-15T15:17:04.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.200+0000 7f34508ba640 1 -- 172.21.15.46:0/2741332244 wait complete. 2024-04-15T15:17:04.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.201+0000 7f34508ba640 1 Processor -- start 2024-04-15T15:17:04.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.201+0000 7f34508ba640 1 -- start start 2024-04-15T15:17:04.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f34508ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f344c0d15e0 0x7f344c1617f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f34508ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f344c0d4a40 0x7f344c15ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f34508ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 0x7f344c14a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f34508ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f344c0d4240 con 0x7f344c15e790 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f34508ba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f344c0d53f0 con 0x7f344c0d4a40 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f34508ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f344c0b7db0 con 0x7f344c0d15e0 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f344ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 0x7f344c14a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f344ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 0x7f344c14a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48126/0 (socket says 172.21.15.46:48126) 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f344ad9d640 1 -- 172.21.15.46:0/4112861165 learned_addr learned my addr 172.21.15.46:0/4112861165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f3449d9b640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f344c0d4a40 0x7f344c15ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f344a59c640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f344c0d15e0 0x7f344c1617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f344ad9d640 1 -- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f344c0d15e0 msgr2=0x7f344c1617f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.202+0000 7f344ad9d640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f344c0d15e0 0x7f344c1617f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f344ad9d640 1 -- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f344c0d4a40 msgr2=0x7f344c15ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f344ad9d640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f344c0d4a40 0x7f344c15ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f344ad9d640 1 -- 172.21.15.46:0/4112861165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c15f570 con 0x7f344c15e790 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f3449d9b640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f344c0d4a40 0x7f344c15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f344ad9d640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 0x7f344c14a980 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f344400b440 tx=0x7f344400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444015020 con 0x7f344c15e790 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3444011830 con 0x7f344c15e790 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444011c20 con 0x7f344c15e790 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c15f850 con 0x7f344c15e790 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.203+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c15fe10 con 0x7f344c15e790 2024-04-15T15:17:04.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.204+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c0689e0 con 0x7f344c15e790 2024-04-15T15:17:04.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.204+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3444018020 con 0x7f344c15e790 2024-04-15T15:17:04.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.205+0000 7f3442ffd640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f342c06c8f0 0x7f342c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.205+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f344408bb50 con 0x7f344c15e790 2024-04-15T15:17:04.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.205+0000 7f344a59c640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f342c06c8f0 0x7f342c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.206+0000 7f344a59c640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f342c06c8f0 0x7f342c06edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f343c001db0 tx=0x7f343c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.210+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f344401f020 con 0x7f344c15e790 2024-04-15T15:17:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.436+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f344c0ce4f0 con 0x7f344c15e790 2024-04-15T15:17:04.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.436+0000 7f3442ffd640 1 -- 172.21.15.46:0/4112861165 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5175 (secure 0 0 0) 0x7f34440172c0 con 0x7f344c15e790 2024-04-15T15:17:04.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:04.439 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:03.437143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5454":{"gid":5454,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/964535021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":964535021},{"type":"v1","addr":"172.21.15.46:6835","nonce":964535021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5454,"qdb_cluster":[5454]},"id":2}]} 2024-04-15T15:17:04.439 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-04-15T15:17:04.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f342c06c8f0 msgr2=0x7f342c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f342c06c8f0 0x7f342c06edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f343c001db0 tx=0x7f343c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 msgr2=0x7f344c14a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 0x7f344c14a980 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f344400b440 tx=0x7f344400b900 comp rx=0 tx=0).stop 2024-04-15T15:17:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 shutdown_connections 2024-04-15T15:17:04.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f342c06c8f0 0x7f342c06edb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f344c15e790 0x7f344c14a980 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f344c0d4a40 0x7f344c15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 --2- 172.21.15.46:0/4112861165 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f344c0d15e0 0x7f344c1617f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 >> 172.21.15.46:0/4112861165 conn(0x7f344c067b40 msgr2=0x7f344c0b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 shutdown_connections 2024-04-15T15:17:04.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.439+0000 7f34508ba640 1 -- 172.21.15.46:0/4112861165 wait complete. 2024-04-15T15:17:04.452 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:04.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.556+0000 7f8ab17f4640 1 Processor -- start 2024-04-15T15:17:04.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.556+0000 7f8ab17f4640 1 -- start start 2024-04-15T15:17:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8ab17f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8aac0b1480 0x7f8aac0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8ab17f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8aac0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8ab17f4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aac0d0190 con 0x7f8aac15fda0 2024-04-15T15:17:04.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8ab17f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f8aac0b1f40 con 0x7f8aac0b1480 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8ab17f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac0cda30 con 0x7f8aac0cf800 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8aac15fda0 0x7f8aac0ce7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52234/0 (socket says 172.21.15.46:52234) 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaaffd640 1 -- 172.21.15.46:0/3374877368 learned_addr learned my addr 172.21.15.46:0/3374877368 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aab7fe640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8aac0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aa3fff640 1 --2- 172.21.15.46:0/3374877368 >> v2:172.21.15.78:3300/0 conn(0x7f8aac0b1480 0x7f8aac0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2353516262 0 0) 0x7f8aac0d0190 con 0x7f8aac15fda0 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a98003520 con 0x7f8aac15fda0 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2245308136 0 0) 0x7f8a98003520 con 0x7f8aac15fda0 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 msgr2=0x7f8aac0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8aac0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 >> v2:172.21.15.78:3300/0 conn(0x7f8aac0b1480 msgr2=0x7f8aac0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 --2- 172.21.15.46:0/3374877368 >> v2:172.21.15.78:3300/0 conn(0x7f8aac0b1480 0x7f8aac0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac0cdd10 con 0x7f8aac15fda0 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f8a9c001670 con 0x7f8aac15fda0 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 >> v1:172.21.15.46:6789/0 conn(0x7f8aac15fda0 legacy=0x7f8aac0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a980039b0 0x7f8a98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a98004800 0x7f8a98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8a980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0b1f40 con 0x7f8a980039b0 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0cda30 con 0x7f8a98004800 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9800b9c0 con 0x7f8aac0cf800 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.557+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a980039b0 0x7f8a98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aa3fff640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a98004800 0x7f8a98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aab7fe640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8a980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aaaffd640 1 -- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 msgr2=0x7f8a980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8a980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aaaffd640 1 -- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a98004800 msgr2=0x7f8a98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a98004800 0x7f8a98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aaaffd640 1 -- 172.21.15.46:0/3374877368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a98003520 con 0x7f8a980039b0 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.558+0000 7f8aab7fe640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8aac0cf800 0x7f8a980051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a980039b0 0x7f8a98003dd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c007fd0 tx=0x7f8a98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a44e0e71700387d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c007d50 con 0x7f8a980039b0 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9c030070 con 0x7f8a980039b0 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8aaa7fc640 1 -- 172.21.15.46:0/3374877368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c02cd40 con 0x7f8a980039b0 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8ab17f4640 1 -- 172.21.15.46:0/3374877368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a980039b0 msgr2=0x7f8a98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/3374877368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a980039b0 0x7f8a98003dd0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c007fd0 tx=0x7f8a98005830 comp rx=0 tx=0).stop 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8aab7fe640 1 -- 172.21.15.46:0/3374877368 reap_dead start 2024-04-15T15:17:04.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8ab17f4640 1 -- 172.21.15.46:0/3374877368 shutdown_connections 2024-04-15T15:17:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8ab17f4640 1 -- 172.21.15.46:0/3374877368 >> 172.21.15.46:0/3374877368 conn(0x7f8aac0b6870 msgr2=0x7f8aac0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8ab17f4640 1 -- 172.21.15.46:0/3374877368 shutdown_connections 2024-04-15T15:17:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.559+0000 7f8ab17f4640 1 -- 172.21.15.46:0/3374877368 wait complete. 2024-04-15T15:17:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.560+0000 7f8ab17f4640 1 Processor -- start 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 -- start start 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a980039b0 0x7f8aac0ccbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8aac149cb0 0x7f8aac070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 0x7f8aac070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac160180 con 0x7f8aac14a110 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0ce2d0 con 0x7f8aac149cb0 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8ab17f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac0cef00 con 0x7f8a980039b0 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8aab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 0x7f8aac070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8aab7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 0x7f8aac070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48146/0 (socket says 172.21.15.46:48146) 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.561+0000 7f8aab7fe640 1 -- 172.21.15.46:0/4137515022 learned_addr learned my addr 172.21.15.46:0/4137515022 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a980039b0 0x7f8aac0ccbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aa3fff640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8aac149cb0 0x7f8aac070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aab7fe640 1 -- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a980039b0 msgr2=0x7f8aac0ccbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aab7fe640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a980039b0 0x7f8aac0ccbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aab7fe640 1 -- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8aac149cb0 msgr2=0x7f8aac070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aab7fe640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8aac149cb0 0x7f8aac070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aab7fe640 1 -- 172.21.15.46:0/4137515022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac0cd110 con 0x7f8aac14a110 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a980039b0 0x7f8aac0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aab7fe640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 0x7f8aac070680 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8aac06e6b0 tx=0x7f8aa4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4010da0 con 0x7f8aac14a110 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa4003990 con 0x7f8aac14a110 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4003d80 con 0x7f8aac14a110 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac1591d0 con 0x7f8aac14a110 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.562+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac159790 con 0x7f8aac14a110 2024-04-15T15:17:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.563+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac0689e0 con 0x7f8aac14a110 2024-04-15T15:17:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.564+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8aa401e020 con 0x7f8aac14a110 2024-04-15T15:17:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.564+0000 7f8aa37fe640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8a8c06c920 0x7f8a8c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.564+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8aa408c330 con 0x7f8aac14a110 2024-04-15T15:17:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.564+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8a8c06c920 0x7f8a8c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.565+0000 7f8aaaffd640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8a8c06c920 0x7f8a8c06ede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c034000 tx=0x7f8a9c007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.569+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8aa405a540 con 0x7f8aac14a110 2024-04-15T15:17:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.756+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8aac0d19c0 con 0x7f8aac14a110 2024-04-15T15:17:04.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.758+0000 7f8aa37fe640 1 -- 172.21.15.46:0/4137515022 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7f8aa4059e40 con 0x7f8aac14a110 2024-04-15T15:17:04.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:04.760 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:02.054907+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8a8c06c920 msgr2=0x7f8a8c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8a8c06c920 0x7f8a8c06ede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c034000 tx=0x7f8a9c007000 comp rx=0 tx=0).stop 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 msgr2=0x7f8aac070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 0x7f8aac070680 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8aac06e6b0 tx=0x7f8aa4010590 comp rx=0 tx=0).stop 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 shutdown_connections 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8a8c06c920 0x7f8a8c06ede0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8aac14a110 0x7f8aac070680 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8aac149cb0 0x7f8aac070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 --2- 172.21.15.46:0/4137515022 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a980039b0 0x7f8aac0ccbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 >> 172.21.15.46:0/4137515022 conn(0x7f8aac0b6870 msgr2=0x7f8aac15d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 shutdown_connections 2024-04-15T15:17:04.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.760+0000 7f8ab17f4640 1 -- 172.21.15.46:0/4137515022 wait complete. 2024-04-15T15:17:04.774 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:17:04.774 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.877+0000 7fe3f3e28640 1 Processor -- start 2024-04-15T15:17:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.877+0000 7fe3f3e28640 1 -- start start 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f3e28640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec0d3240 0x7fe3ec0d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f3e28640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3ec0d1440 0x7fe3ec0b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f3e28640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3ec0d2480 con 0x7fe3ec15e790 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f3e28640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe3ec0d2db0 con 0x7fe3ec0d1440 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f3e28640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec0d2760 con 0x7fe3ec0d3240 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f23c5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3ec0d1440 0x7fe3ec0b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f13c3640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec0d3240 0x7fe3ec0d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f23c5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe3ec0d1440 0x7fe3ec0b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:55826/0 (socket says 172.21.15.46:55826) 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f23c5640 1 -- 172.21.15.46:0/3623767750 learned_addr learned my addr 172.21.15.46:0/3623767750 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f13c3640 1 --2- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec0d3240 0x7fe3ec0d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:59866/0 (socket says 172.21.15.46:59866) 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f1bc4640 1 --1- 172.21.15.46:0/3623767750 >> v1:172.21.15.78:6789/0 conn(0x7fe3ec15e790 0x7fe3ec0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:51062/0 (socket says 172.21.15.46:51062) 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f23c5640 1 -- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec0d3240 msgr2=0x7fe3ec0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f23c5640 1 --2- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec0d3240 0x7fe3ec0d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.878+0000 7fe3f23c5640 1 -- 172.21.15.46:0/3623767750 >> v1:172.21.15.78:6789/0 conn(0x7fe3ec15e790 legacy=0x7fe3ec0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.879+0000 7fe3f23c5640 1 -- 172.21.15.46:0/3623767750 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec0d2a40 con 0x7fe3ec0d1440 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.879+0000 7fe3f23c5640 1 --2- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0b74f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009e50 tx=0x7fe3e402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baf14c3303c0ecb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.879+0000 7fe3f0bc2640 1 -- 172.21.15.46:0/3623767750 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e402cb00 con 0x7fe3ec0d1440 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.879+0000 7fe3f0bc2640 1 -- 172.21.15.46:0/3623767750 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e402cca0 con 0x7fe3ec0d1440 2024-04-15T15:17:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f0bc2640 1 -- 172.21.15.46:0/3623767750 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4034610 con 0x7fe3ec0d1440 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 -- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d1440 msgr2=0x7fe3ec0b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0b74f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009e50 tx=0x7fe3e402bf60 comp rx=0 tx=0).stop 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 -- 172.21.15.46:0/3623767750 shutdown_connections 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0b74f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/3623767750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec0d3240 0x7fe3ec0d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 -- 172.21.15.46:0/3623767750 >> 172.21.15.46:0/3623767750 conn(0x7fe3ec08bf60 msgr2=0x7fe3ec0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 -- 172.21.15.46:0/3623767750 shutdown_connections 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.880+0000 7fe3f3e28640 1 -- 172.21.15.46:0/3623767750 wait complete. 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.881+0000 7fe3f3e28640 1 Processor -- start 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.881+0000 7fe3f3e28640 1 -- start start 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f3e28640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f3e28640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d3240 0x7fe3ec0b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f3e28640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec15e790 0x7fe3ec1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f3e28640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec0d5180 con 0x7fe3ec0d1440 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f3e28640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec0d0430 con 0x7fe3ec0d3240 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f3e28640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe3ec0d5300 con 0x7fe3ec15e790 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48156/0 (socket says 172.21.15.46:48156) 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 -- 172.21.15.46:0/2575485134 learned_addr learned my addr 172.21.15.46:0/2575485134 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f13c3640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d3240 0x7fe3ec0b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f23c5640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec15e790 0x7fe3ec1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 -- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec15e790 msgr2=0x7fe3ec1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec15e790 0x7fe3ec1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 -- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d3240 msgr2=0x7fe3ec0b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.882+0000 7fe3f1bc4640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d3240 0x7fe3ec0b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3f1bc4640 1 -- 172.21.15.46:0/2575485134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec0ccb30 con 0x7fe3ec0d1440 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3f13c3640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d3240 0x7fe3ec0b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3f1bc4640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0c9fb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800c3f0 tx=0x7fe3e800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e80074e0 con 0x7fe3ec0d1440 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e8007680 con 0x7fe3ec0d1440 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec0cce10 con 0x7fe3ec0d1440 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8004a00 con 0x7fe3ec0d1440 2024-04-15T15:17:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.883+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec0cd6b0 con 0x7fe3ec0d1440 2024-04-15T15:17:04.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.884+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec0ccfe0 con 0x7fe3ec0d1440 2024-04-15T15:17:04.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.884+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe3e801a020 con 0x7fe3ec0d1440 2024-04-15T15:17:04.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.885+0000 7fe3e27fc640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe3d006c8f0 0x7fe3d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.885+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe3e808b4a0 con 0x7fe3ec0d1440 2024-04-15T15:17:04.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.885+0000 7fe3f13c3640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe3d006c8f0 0x7fe3d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.886+0000 7fe3f13c3640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe3d006c8f0 0x7fe3d006edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0b5300 tx=0x7fe3dc008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:04.889+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3e8059740 con 0x7fe3ec0d1440 2024-04-15T15:17:05.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.069+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe3ec06ea90 con 0x7fe3d006c8f0 2024-04-15T15:17:05.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.077+0000 7fe3e27fc640 1 -- 172.21.15.46:0/2575485134 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248614 (secure 0 0 0) 0x7fe3ec06ea90 con 0x7fe3d006c8f0 2024-04-15T15:17:05.082 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:05.083 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:05.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe3d006c8f0 msgr2=0x7fe3d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe3d006c8f0 0x7fe3d006edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec0b5300 tx=0x7fe3dc008040 comp rx=0 tx=0).stop 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 msgr2=0x7fe3ec0c9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0c9fb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800c3f0 tx=0x7fe3e800c8b0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 shutdown_connections 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe3d006c8f0 0x7fe3d006edb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe3ec15e790 0x7fe3ec1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe3ec0d3240 0x7fe3ec0b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 --2- 172.21.15.46:0/2575485134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec0d1440 0x7fe3ec0c9fb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 >> 172.21.15.46:0/2575485134 conn(0x7fe3ec08bf60 msgr2=0x7fe3ec0b7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 shutdown_connections 2024-04-15T15:17:05.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.080+0000 7fe3f3e28640 1 -- 172.21.15.46:0/2575485134 wait complete. 2024-04-15T15:17:05.092 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-15T15:17:04.373266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237304,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737416,"statfs":{"total":773094113280,"available":772851113984,"internally_reserved":0,"allocated":4595712,"data_stored":2630137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.338094"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048962+0000","last_change":"2024-04-15T15:17:00.048962+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048962+0000","last_undegraded":"2024-04-15T15:17:00.048962+0000","last_fullsized":"2024-04-15T15:17:00.048962+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046889+0000","last_change":"2024-04-15T15:17:00.046889+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046889+0000","last_undegraded":"2024-04-15T15:17:00.046889+0000","last_fullsized":"2024-04-15T15:17:00.046889+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047885+0000","last_change":"2024-04-15T15:17:00.047885+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047885+0000","last_undegraded":"2024-04-15T15:17:00.047885+0000","last_fullsized":"2024-04-15T15:17:00.047885+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045994+0000","last_change":"2024-04-15T15:17:00.045994+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045994+0000","last_undegraded":"2024-04-15T15:17:00.045994+0000","last_fullsized":"2024-04-15T15:17:00.045994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043764+0000","last_change":"2024-04-15T15:17:00.043764+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043764+0000","last_undegraded":"2024-04-15T15:17:00.043764+0000","last_fullsized":"2024-04-15T15:17:00.043764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044975+0000","last_change":"2024-04-15T15:17:00.044975+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044975+0000","last_undegraded":"2024-04-15T15:17:00.044975+0000","last_fullsized":"2024-04-15T15:17:00.044975+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074681+0000","last_change":"2024-04-15T15:17:01.069002+0000","last_active":"2024-04-15T15:17:01.074681+0000","last_peered":"2024-04-15T15:17:01.074681+0000","last_clean":"2024-04-15T15:17:01.074681+0000","last_became_active":"2024-04-15T15:17:01.068755+0000","last_became_peered":"2024-04-15T15:17:01.068755+0000","last_unstale":"2024-04-15T15:17:01.074681+0000","last_undegraded":"2024-04-15T15:17:01.074681+0000","last_fullsized":"2024-04-15T15:17:01.074681+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.049417+0000","last_change":"2024-04-15T15:17:00.049417+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.049417+0000","last_undegraded":"2024-04-15T15:17:00.049417+0000","last_fullsized":"2024-04-15T15:17:00.049417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.049028+0000","last_change":"2024-04-15T15:17:00.049028+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.049028+0000","last_undegraded":"2024-04-15T15:17:00.049028+0000","last_fullsized":"2024-04-15T15:17:00.049028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073089+0000","last_change":"2024-04-15T15:17:01.068258+0000","last_active":"2024-04-15T15:17:01.073089+0000","last_peered":"2024-04-15T15:17:01.073089+0000","last_clean":"2024-04-15T15:17:01.073089+0000","last_became_active":"2024-04-15T15:17:01.067888+0000","last_became_peered":"2024-04-15T15:17:01.067888+0000","last_unstale":"2024-04-15T15:17:01.073089+0000","last_undegraded":"2024-04-15T15:17:01.073089+0000","last_fullsized":"2024-04-15T15:17:01.073089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075397+0000","last_change":"2024-04-15T15:17:01.070976+0000","last_active":"2024-04-15T15:17:01.075397+0000","last_peered":"2024-04-15T15:17:01.075397+0000","last_clean":"2024-04-15T15:17:01.075397+0000","last_became_active":"2024-04-15T15:17:01.070678+0000","last_became_peered":"2024-04-15T15:17:01.070678+0000","last_unstale":"2024-04-15T15:17:01.075397+0000","last_undegraded":"2024-04-15T15:17:01.075397+0000","last_fullsized":"2024-04-15T15:17:01.075397+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:38:18.560062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045186+0000","last_change":"2024-04-15T15:17:00.045186+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045186+0000","last_undegraded":"2024-04-15T15:17:00.045186+0000","last_fullsized":"2024-04-15T15:17:00.045186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.065827+0000","last_change":"2024-04-15T15:17:01.064962+0000","last_active":"2024-04-15T15:17:01.065827+0000","last_peered":"2024-04-15T15:17:01.065827+0000","last_clean":"2024-04-15T15:17:01.065827+0000","last_became_active":"2024-04-15T15:17:01.064395+0000","last_became_peered":"2024-04-15T15:17:01.064395+0000","last_unstale":"2024-04-15T15:17:01.065827+0000","last_undegraded":"2024-04-15T15:17:01.065827+0000","last_fullsized":"2024-04-15T15:17:01.065827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072393+0000","last_change":"2024-04-15T15:17:01.069059+0000","last_active":"2024-04-15T15:17:01.072393+0000","last_peered":"2024-04-15T15:17:01.072393+0000","last_clean":"2024-04-15T15:17:01.072393+0000","last_became_active":"2024-04-15T15:17:01.067835+0000","last_became_peered":"2024-04-15T15:17:01.067835+0000","last_unstale":"2024-04-15T15:17:01.072393+0000","last_undegraded":"2024-04-15T15:17:01.072393+0000","last_fullsized":"2024-04-15T15:17:01.072393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073052+0000","last_change":"2024-04-15T15:17:01.065939+0000","last_active":"2024-04-15T15:17:01.073052+0000","last_peered":"2024-04-15T15:17:01.073052+0000","last_clean":"2024-04-15T15:17:01.073052+0000","last_became_active":"2024-04-15T15:17:01.065641+0000","last_became_peered":"2024-04-15T15:17:01.065641+0000","last_unstale":"2024-04-15T15:17:01.073052+0000","last_undegraded":"2024-04-15T15:17:01.073052+0000","last_fullsized":"2024-04-15T15:17:01.073052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052205+0000","last_change":"2024-04-15T15:17:00.052205+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052205+0000","last_undegraded":"2024-04-15T15:17:00.052205+0000","last_fullsized":"2024-04-15T15:17:00.052205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047966+0000","last_change":"2024-04-15T15:17:00.047966+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047966+0000","last_undegraded":"2024-04-15T15:17:00.047966+0000","last_fullsized":"2024-04-15T15:17:00.047966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073211+0000","last_change":"2024-04-15T15:17:01.070470+0000","last_active":"2024-04-15T15:17:01.073211+0000","last_peered":"2024-04-15T15:17:01.073211+0000","last_clean":"2024-04-15T15:17:01.073211+0000","last_became_active":"2024-04-15T15:17:01.070200+0000","last_became_peered":"2024-04-15T15:17:01.070200+0000","last_unstale":"2024-04-15T15:17:01.073211+0000","last_undegraded":"2024-04-15T15:17:01.073211+0000","last_fullsized":"2024-04-15T15:17:01.073211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043158+0000","last_change":"2024-04-15T15:17:00.043158+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043158+0000","last_undegraded":"2024-04-15T15:17:00.043158+0000","last_fullsized":"2024-04-15T15:17:00.043158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.076558+0000","last_change":"2024-04-15T15:17:01.071687+0000","last_active":"2024-04-15T15:17:01.076558+0000","last_peered":"2024-04-15T15:17:01.076558+0000","last_clean":"2024-04-15T15:17:01.076558+0000","last_became_active":"2024-04-15T15:17:01.071330+0000","last_became_peered":"2024-04-15T15:17:01.071330+0000","last_unstale":"2024-04-15T15:17:01.076558+0000","last_undegraded":"2024-04-15T15:17:01.076558+0000","last_fullsized":"2024-04-15T15:17:01.076558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073107+0000","last_change":"2024-04-15T15:17:01.067759+0000","last_active":"2024-04-15T15:17:01.073107+0000","last_peered":"2024-04-15T15:17:01.073107+0000","last_clean":"2024-04-15T15:17:01.073107+0000","last_became_active":"2024-04-15T15:17:01.067476+0000","last_became_peered":"2024-04-15T15:17:01.067476+0000","last_unstale":"2024-04-15T15:17:01.073107+0000","last_undegraded":"2024-04-15T15:17:01.073107+0000","last_fullsized":"2024-04-15T15:17:01.073107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072415+0000","last_change":"2024-04-15T15:17:01.070298+0000","last_active":"2024-04-15T15:17:01.072415+0000","last_peered":"2024-04-15T15:17:01.072415+0000","last_clean":"2024-04-15T15:17:01.072415+0000","last_became_active":"2024-04-15T15:17:01.070032+0000","last_became_peered":"2024-04-15T15:17:01.070032+0000","last_unstale":"2024-04-15T15:17:01.072415+0000","last_undegraded":"2024-04-15T15:17:01.072415+0000","last_fullsized":"2024-04-15T15:17:01.072415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:23.034253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052838+0000","last_change":"2024-04-15T15:17:00.052838+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052838+0000","last_undegraded":"2024-04-15T15:17:00.052838+0000","last_fullsized":"2024-04-15T15:17:00.052838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.042458+0000","last_change":"2024-04-15T15:17:00.042458+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.042458+0000","last_undegraded":"2024-04-15T15:17:00.042458+0000","last_fullsized":"2024-04-15T15:17:00.042458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061210+0000","last_change":"2024-04-15T15:17:01.056648+0000","last_active":"2024-04-15T15:17:01.061210+0000","last_peered":"2024-04-15T15:17:01.061210+0000","last_clean":"2024-04-15T15:17:01.061210+0000","last_became_active":"2024-04-15T15:17:01.056354+0000","last_became_peered":"2024-04-15T15:17:01.056354+0000","last_unstale":"2024-04-15T15:17:01.061210+0000","last_undegraded":"2024-04-15T15:17:01.061210+0000","last_fullsized":"2024-04-15T15:17:01.061210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044860+0000","last_change":"2024-04-15T15:17:00.044860+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044860+0000","last_undegraded":"2024-04-15T15:17:00.044860+0000","last_fullsized":"2024-04-15T15:17:00.044860+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.062082+0000","last_change":"2024-04-15T15:17:01.059015+0000","last_active":"2024-04-15T15:17:01.062082+0000","last_peered":"2024-04-15T15:17:01.062082+0000","last_clean":"2024-04-15T15:17:01.062082+0000","last_became_active":"2024-04-15T15:17:01.058718+0000","last_became_peered":"2024-04-15T15:17:01.058718+0000","last_unstale":"2024-04-15T15:17:01.062082+0000","last_undegraded":"2024-04-15T15:17:01.062082+0000","last_fullsized":"2024-04-15T15:17:01.062082+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074673+0000","last_change":"2024-04-15T15:17:01.069694+0000","last_active":"2024-04-15T15:17:01.074673+0000","last_peered":"2024-04-15T15:17:01.074673+0000","last_clean":"2024-04-15T15:17:01.074673+0000","last_became_active":"2024-04-15T15:17:01.069408+0000","last_became_peered":"2024-04-15T15:17:01.069408+0000","last_unstale":"2024-04-15T15:17:01.074673+0000","last_undegraded":"2024-04-15T15:17:01.074673+0000","last_fullsized":"2024-04-15T15:17:01.074673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072620+0000","last_change":"2024-04-15T15:17:01.068919+0000","last_active":"2024-04-15T15:17:01.072620+0000","last_peered":"2024-04-15T15:17:01.072620+0000","last_clean":"2024-04-15T15:17:01.072620+0000","last_became_active":"2024-04-15T15:17:01.068639+0000","last_became_peered":"2024-04-15T15:17:01.068639+0000","last_unstale":"2024-04-15T15:17:01.072620+0000","last_undegraded":"2024-04-15T15:17:01.072620+0000","last_fullsized":"2024-04-15T15:17:01.072620+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061418+0000","last_change":"2024-04-15T15:17:01.056050+0000","last_active":"2024-04-15T15:17:01.061418+0000","last_peered":"2024-04-15T15:17:01.061418+0000","last_clean":"2024-04-15T15:17:01.061418+0000","last_became_active":"2024-04-15T15:17:01.055735+0000","last_became_peered":"2024-04-15T15:17:01.055735+0000","last_unstale":"2024-04-15T15:17:01.061418+0000","last_undegraded":"2024-04-15T15:17:01.061418+0000","last_fullsized":"2024-04-15T15:17:01.061418+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:51.299505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048125+0000","last_change":"2024-04-15T15:17:00.048125+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048125+0000","last_undegraded":"2024-04-15T15:17:00.048125+0000","last_fullsized":"2024-04-15T15:17:00.048125+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044106+0000","last_change":"2024-04-15T15:17:00.044106+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044106+0000","last_undegraded":"2024-04-15T15:17:00.044106+0000","last_fullsized":"2024-04-15T15:17:00.044106+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047283+0000","last_change":"2024-04-15T15:17:00.047283+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047283+0000","last_undegraded":"2024-04-15T15:17:00.047283+0000","last_fullsized":"2024-04-15T15:17:00.047283+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078586+0000","last_change":"2024-04-15T15:17:00.053485+0000","last_active":"2024-04-15T15:17:01.078586+0000","last_peered":"2024-04-15T15:17:01.078586+0000","last_clean":"2024-04-15T15:17:01.078586+0000","last_became_active":"2024-04-15T15:17:00.053211+0000","last_became_peered":"2024-04-15T15:17:00.053211+0000","last_unstale":"2024-04-15T15:17:01.078586+0000","last_undegraded":"2024-04-15T15:17:01.078586+0000","last_fullsized":"2024-04-15T15:17:01.078586+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043633+0000","last_change":"2024-04-15T15:17:00.043633+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043633+0000","last_undegraded":"2024-04-15T15:17:00.043633+0000","last_fullsized":"2024-04-15T15:17:00.043633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058716+0000","last_change":"2024-04-15T15:17:00.056209+0000","last_active":"2024-04-15T15:17:00.058716+0000","last_peered":"2024-04-15T15:17:00.058716+0000","last_clean":"2024-04-15T15:17:00.058716+0000","last_became_active":"2024-04-15T15:17:00.055881+0000","last_became_peered":"2024-04-15T15:17:00.055881+0000","last_unstale":"2024-04-15T15:17:00.058716+0000","last_undegraded":"2024-04-15T15:17:00.058716+0000","last_fullsized":"2024-04-15T15:17:00.058716+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072065+0000","last_change":"2024-04-15T15:17:01.067226+0000","last_active":"2024-04-15T15:17:01.072065+0000","last_peered":"2024-04-15T15:17:01.072065+0000","last_clean":"2024-04-15T15:17:01.072065+0000","last_became_active":"2024-04-15T15:17:01.066479+0000","last_became_peered":"2024-04-15T15:17:01.066479+0000","last_unstale":"2024-04-15T15:17:01.072065+0000","last_undegraded":"2024-04-15T15:17:01.072065+0000","last_fullsized":"2024-04-15T15:17:01.072065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058483+0000","last_change":"2024-04-15T15:17:00.054251+0000","last_active":"2024-04-15T15:17:00.058483+0000","last_peered":"2024-04-15T15:17:00.058483+0000","last_clean":"2024-04-15T15:17:00.058483+0000","last_became_active":"2024-04-15T15:17:00.053980+0000","last_became_peered":"2024-04-15T15:17:00.053980+0000","last_unstale":"2024-04-15T15:17:00.058483+0000","last_undegraded":"2024-04-15T15:17:00.058483+0000","last_fullsized":"2024-04-15T15:17:00.058483+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.050290+0000","last_change":"2024-04-15T15:17:00.050290+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.050290+0000","last_undegraded":"2024-04-15T15:17:00.050290+0000","last_fullsized":"2024-04-15T15:17:00.050290+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.080694+0000","last_change":"2024-04-15T15:17:00.058378+0000","last_active":"2024-04-15T15:17:01.080694+0000","last_peered":"2024-04-15T15:17:01.080694+0000","last_clean":"2024-04-15T15:17:01.080694+0000","last_became_active":"2024-04-15T15:17:00.058024+0000","last_became_peered":"2024-04-15T15:17:00.058024+0000","last_unstale":"2024-04-15T15:17:01.080694+0000","last_undegraded":"2024-04-15T15:17:01.080694+0000","last_fullsized":"2024-04-15T15:17:01.080694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047108+0000","last_change":"2024-04-15T15:17:00.047108+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047108+0000","last_undegraded":"2024-04-15T15:17:00.047108+0000","last_fullsized":"2024-04-15T15:17:00.047108+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.077676+0000","last_change":"2024-04-15T15:17:00.052588+0000","last_active":"2024-04-15T15:17:01.077676+0000","last_peered":"2024-04-15T15:17:01.077676+0000","last_clean":"2024-04-15T15:17:01.077676+0000","last_became_active":"2024-04-15T15:17:00.052287+0000","last_became_peered":"2024-04-15T15:17:00.052287+0000","last_unstale":"2024-04-15T15:17:01.077676+0000","last_undegraded":"2024-04-15T15:17:01.077676+0000","last_fullsized":"2024-04-15T15:17:01.077676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079604+0000","last_change":"2024-04-15T15:17:01.077529+0000","last_active":"2024-04-15T15:17:01.079604+0000","last_peered":"2024-04-15T15:17:01.079604+0000","last_clean":"2024-04-15T15:17:01.079604+0000","last_became_active":"2024-04-15T15:17:01.077231+0000","last_became_peered":"2024-04-15T15:17:01.077231+0000","last_unstale":"2024-04-15T15:17:01.079604+0000","last_undegraded":"2024-04-15T15:17:01.079604+0000","last_fullsized":"2024-04-15T15:17:01.079604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:58:51.072457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.063827+0000","last_change":"2024-04-15T15:17:00.060924+0000","last_active":"2024-04-15T15:17:00.063827+0000","last_peered":"2024-04-15T15:17:00.063827+0000","last_clean":"2024-04-15T15:17:00.063827+0000","last_became_active":"2024-04-15T15:17:00.060610+0000","last_became_peered":"2024-04-15T15:17:00.060610+0000","last_unstale":"2024-04-15T15:17:00.063827+0000","last_undegraded":"2024-04-15T15:17:00.063827+0000","last_fullsized":"2024-04-15T15:17:00.063827+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072370+0000","last_change":"2024-04-15T15:17:01.068809+0000","last_active":"2024-04-15T15:17:01.072370+0000","last_peered":"2024-04-15T15:17:01.072370+0000","last_clean":"2024-04-15T15:17:01.072370+0000","last_became_active":"2024-04-15T15:17:01.067740+0000","last_became_peered":"2024-04-15T15:17:01.067740+0000","last_unstale":"2024-04-15T15:17:01.072370+0000","last_undegraded":"2024-04-15T15:17:01.072370+0000","last_fullsized":"2024-04-15T15:17:01.072370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058130+0000","last_change":"2024-04-15T15:17:00.056079+0000","last_active":"2024-04-15T15:17:00.058130+0000","last_peered":"2024-04-15T15:17:00.058130+0000","last_clean":"2024-04-15T15:17:00.058130+0000","last_became_active":"2024-04-15T15:17:00.055794+0000","last_became_peered":"2024-04-15T15:17:00.055794+0000","last_unstale":"2024-04-15T15:17:00.058130+0000","last_undegraded":"2024-04-15T15:17:00.058130+0000","last_fullsized":"2024-04-15T15:17:00.058130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.050788+0000","last_change":"2024-04-15T15:17:00.050788+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.050788+0000","last_undegraded":"2024-04-15T15:17:00.050788+0000","last_fullsized":"2024-04-15T15:17:00.050788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056679+0000","last_change":"2024-04-15T15:17:00.048501+0000","last_active":"2024-04-15T15:17:01.056679+0000","last_peered":"2024-04-15T15:17:01.056679+0000","last_clean":"2024-04-15T15:17:01.056679+0000","last_became_active":"2024-04-15T15:17:00.048148+0000","last_became_peered":"2024-04-15T15:17:00.048148+0000","last_unstale":"2024-04-15T15:17:01.056679+0000","last_undegraded":"2024-04-15T15:17:01.056679+0000","last_fullsized":"2024-04-15T15:17:01.056679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043571+0000","last_change":"2024-04-15T15:17:00.043571+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043571+0000","last_undegraded":"2024-04-15T15:17:00.043571+0000","last_fullsized":"2024-04-15T15:17:00.043571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.060211+0000","last_change":"2024-04-15T15:17:00.059059+0000","last_active":"2024-04-15T15:17:00.060211+0000","last_peered":"2024-04-15T15:17:00.060211+0000","last_clean":"2024-04-15T15:17:00.060211+0000","last_became_active":"2024-04-15T15:17:00.058527+0000","last_became_peered":"2024-04-15T15:17:00.058527+0000","last_unstale":"2024-04-15T15:17:00.060211+0000","last_undegraded":"2024-04-15T15:17:00.060211+0000","last_fullsized":"2024-04-15T15:17:00.060211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052662+0000","last_change":"2024-04-15T15:17:00.052662+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052662+0000","last_undegraded":"2024-04-15T15:17:00.052662+0000","last_fullsized":"2024-04-15T15:17:00.052662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056094+0000","last_change":"2024-04-15T15:17:00.058594+0000","last_active":"2024-04-15T15:17:01.056094+0000","last_peered":"2024-04-15T15:17:01.056094+0000","last_clean":"2024-04-15T15:17:01.056094+0000","last_became_active":"2024-04-15T15:17:00.058359+0000","last_became_peered":"2024-04-15T15:17:00.058359+0000","last_unstale":"2024-04-15T15:17:01.056094+0000","last_undegraded":"2024-04-15T15:17:01.056094+0000","last_fullsized":"2024-04-15T15:17:01.056094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045457+0000","last_change":"2024-04-15T15:17:00.045457+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045457+0000","last_undegraded":"2024-04-15T15:17:00.045457+0000","last_fullsized":"2024-04-15T15:17:00.045457+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078903+0000","last_change":"2024-04-15T15:17:00.055108+0000","last_active":"2024-04-15T15:17:01.078903+0000","last_peered":"2024-04-15T15:17:01.078903+0000","last_clean":"2024-04-15T15:17:01.078903+0000","last_became_active":"2024-04-15T15:17:00.054818+0000","last_became_peered":"2024-04-15T15:17:00.054818+0000","last_unstale":"2024-04-15T15:17:01.078903+0000","last_undegraded":"2024-04-15T15:17:01.078903+0000","last_fullsized":"2024-04-15T15:17:01.078903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072041+0000","last_change":"2024-04-15T15:17:01.066511+0000","last_active":"2024-04-15T15:17:01.072041+0000","last_peered":"2024-04-15T15:17:01.072041+0000","last_clean":"2024-04-15T15:17:01.072041+0000","last_became_active":"2024-04-15T15:17:01.066201+0000","last_became_peered":"2024-04-15T15:17:01.066201+0000","last_unstale":"2024-04-15T15:17:01.072041+0000","last_undegraded":"2024-04-15T15:17:01.072041+0000","last_fullsized":"2024-04-15T15:17:01.072041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.062191+0000","last_change":"2024-04-15T15:17:00.061109+0000","last_active":"2024-04-15T15:17:00.062191+0000","last_peered":"2024-04-15T15:17:00.062191+0000","last_clean":"2024-04-15T15:17:00.062191+0000","last_became_active":"2024-04-15T15:17:00.060823+0000","last_became_peered":"2024-04-15T15:17:00.060823+0000","last_unstale":"2024-04-15T15:17:00.062191+0000","last_undegraded":"2024-04-15T15:17:00.062191+0000","last_fullsized":"2024-04-15T15:17:00.062191+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046672+0000","last_change":"2024-04-15T15:17:00.046672+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046672+0000","last_undegraded":"2024-04-15T15:17:00.046672+0000","last_fullsized":"2024-04-15T15:17:00.046672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058974+0000","last_change":"2024-04-15T15:17:00.057661+0000","last_active":"2024-04-15T15:17:00.058974+0000","last_peered":"2024-04-15T15:17:00.058974+0000","last_clean":"2024-04-15T15:17:00.058974+0000","last_became_active":"2024-04-15T15:17:00.057376+0000","last_became_peered":"2024-04-15T15:17:00.057376+0000","last_unstale":"2024-04-15T15:17:00.058974+0000","last_undegraded":"2024-04-15T15:17:00.058974+0000","last_fullsized":"2024-04-15T15:17:00.058974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:22.029429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048446+0000","last_change":"2024-04-15T15:17:00.048446+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048446+0000","last_undegraded":"2024-04-15T15:17:00.048446+0000","last_fullsized":"2024-04-15T15:17:00.048446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058358+0000","last_change":"2024-04-15T15:17:00.055785+0000","last_active":"2024-04-15T15:17:00.058358+0000","last_peered":"2024-04-15T15:17:00.058358+0000","last_clean":"2024-04-15T15:17:00.058358+0000","last_became_active":"2024-04-15T15:17:00.055509+0000","last_became_peered":"2024-04-15T15:17:00.055509+0000","last_unstale":"2024-04-15T15:17:00.058358+0000","last_undegraded":"2024-04-15T15:17:00.058358+0000","last_fullsized":"2024-04-15T15:17:00.058358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047149+0000","last_change":"2024-04-15T15:17:00.047149+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047149+0000","last_undegraded":"2024-04-15T15:17:00.047149+0000","last_fullsized":"2024-04-15T15:17:00.047149+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048522+0000","last_change":"2024-04-15T15:17:00.055842+0000","last_active":"2024-04-15T15:17:01.048522+0000","last_peered":"2024-04-15T15:17:01.048522+0000","last_clean":"2024-04-15T15:17:01.048522+0000","last_became_active":"2024-04-15T15:17:00.054053+0000","last_became_peered":"2024-04-15T15:17:00.054053+0000","last_unstale":"2024-04-15T15:17:01.048522+0000","last_undegraded":"2024-04-15T15:17:01.048522+0000","last_fullsized":"2024-04-15T15:17:01.048522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047661+0000","last_change":"2024-04-15T15:17:00.047661+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047661+0000","last_undegraded":"2024-04-15T15:17:00.047661+0000","last_fullsized":"2024-04-15T15:17:00.047661+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064003+0000","last_change":"2024-04-15T15:17:00.061010+0000","last_active":"2024-04-15T15:17:00.064003+0000","last_peered":"2024-04-15T15:17:00.064003+0000","last_clean":"2024-04-15T15:17:00.064003+0000","last_became_active":"2024-04-15T15:17:00.060723+0000","last_became_peered":"2024-04-15T15:17:00.060723+0000","last_unstale":"2024-04-15T15:17:00.064003+0000","last_undegraded":"2024-04-15T15:17:00.064003+0000","last_fullsized":"2024-04-15T15:17:00.064003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.051279+0000","last_change":"2024-04-15T15:17:00.051279+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.051279+0000","last_undegraded":"2024-04-15T15:17:00.051279+0000","last_fullsized":"2024-04-15T15:17:00.051279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078087+0000","last_change":"2024-04-15T15:17:00.053088+0000","last_active":"2024-04-15T15:17:01.078087+0000","last_peered":"2024-04-15T15:17:01.078087+0000","last_clean":"2024-04-15T15:17:01.078087+0000","last_became_active":"2024-04-15T15:17:00.052762+0000","last_became_peered":"2024-04-15T15:17:00.052762+0000","last_unstale":"2024-04-15T15:17:01.078087+0000","last_undegraded":"2024-04-15T15:17:01.078087+0000","last_fullsized":"2024-04-15T15:17:01.078087+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075375+0000","last_change":"2024-04-15T15:17:01.067435+0000","last_active":"2024-04-15T15:17:01.075375+0000","last_peered":"2024-04-15T15:17:01.075375+0000","last_clean":"2024-04-15T15:17:01.075375+0000","last_became_active":"2024-04-15T15:17:01.066970+0000","last_became_peered":"2024-04-15T15:17:01.066970+0000","last_unstale":"2024-04-15T15:17:01.075375+0000","last_undegraded":"2024-04-15T15:17:01.075375+0000","last_fullsized":"2024-04-15T15:17:01.075375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079013+0000","last_change":"2024-04-15T15:17:00.049730+0000","last_active":"2024-04-15T15:17:01.079013+0000","last_peered":"2024-04-15T15:17:01.079013+0000","last_clean":"2024-04-15T15:17:01.079013+0000","last_became_active":"2024-04-15T15:17:00.049472+0000","last_became_peered":"2024-04-15T15:17:00.049472+0000","last_unstale":"2024-04-15T15:17:01.079013+0000","last_undegraded":"2024-04-15T15:17:01.079013+0000","last_fullsized":"2024-04-15T15:17:01.079013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072086+0000","last_change":"2024-04-15T15:17:01.067474+0000","last_active":"2024-04-15T15:17:01.072086+0000","last_peered":"2024-04-15T15:17:01.072086+0000","last_clean":"2024-04-15T15:17:01.072086+0000","last_became_active":"2024-04-15T15:17:01.066802+0000","last_became_peered":"2024-04-15T15:17:01.066802+0000","last_unstale":"2024-04-15T15:17:01.072086+0000","last_undegraded":"2024-04-15T15:17:01.072086+0000","last_fullsized":"2024-04-15T15:17:01.072086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.055389+0000","last_change":"2024-04-15T15:17:00.054252+0000","last_active":"2024-04-15T15:17:00.055389+0000","last_peered":"2024-04-15T15:17:00.055389+0000","last_clean":"2024-04-15T15:17:00.055389+0000","last_became_active":"2024-04-15T15:17:00.053917+0000","last_became_peered":"2024-04-15T15:17:00.053917+0000","last_unstale":"2024-04-15T15:17:00.055389+0000","last_undegraded":"2024-04-15T15:17:00.055389+0000","last_fullsized":"2024-04-15T15:17:00.055389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:53:53.053021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043001+0000","last_change":"2024-04-15T15:17:00.043001+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043001+0000","last_undegraded":"2024-04-15T15:17:00.043001+0000","last_fullsized":"2024-04-15T15:17:00.043001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.057998+0000","last_change":"2024-04-15T15:17:00.053573+0000","last_active":"2024-04-15T15:17:00.057998+0000","last_peered":"2024-04-15T15:17:00.057998+0000","last_clean":"2024-04-15T15:17:00.057998+0000","last_became_active":"2024-04-15T15:17:00.053296+0000","last_became_peered":"2024-04-15T15:17:00.053296+0000","last_unstale":"2024-04-15T15:17:00.057998+0000","last_undegraded":"2024-04-15T15:17:00.057998+0000","last_fullsized":"2024-04-15T15:17:00.057998+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047704+0000","last_change":"2024-04-15T15:17:00.047704+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047704+0000","last_undegraded":"2024-04-15T15:17:00.047704+0000","last_fullsized":"2024-04-15T15:17:00.047704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.055352+0000","last_change":"2024-04-15T15:17:00.052685+0000","last_active":"2024-04-15T15:17:00.055352+0000","last_peered":"2024-04-15T15:17:00.055352+0000","last_clean":"2024-04-15T15:17:00.055352+0000","last_became_active":"2024-04-15T15:17:00.052398+0000","last_became_peered":"2024-04-15T15:17:00.052398+0000","last_unstale":"2024-04-15T15:17:00.055352+0000","last_undegraded":"2024-04-15T15:17:00.055352+0000","last_fullsized":"2024-04-15T15:17:00.055352+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:42:40.378774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048032+0000","last_change":"2024-04-15T15:17:00.048032+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048032+0000","last_undegraded":"2024-04-15T15:17:00.048032+0000","last_fullsized":"2024-04-15T15:17:00.048032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.046519+0000","last_change":"2024-04-15T15:17:00.044538+0000","last_active":"2024-04-15T15:17:00.046519+0000","last_peered":"2024-04-15T15:17:00.046519+0000","last_clean":"2024-04-15T15:17:00.046519+0000","last_became_active":"2024-04-15T15:17:00.044237+0000","last_became_peered":"2024-04-15T15:17:00.044237+0000","last_unstale":"2024-04-15T15:17:00.046519+0000","last_undegraded":"2024-04-15T15:17:00.046519+0000","last_fullsized":"2024-04-15T15:17:00.046519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:42.468987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073110+0000","last_change":"2024-04-15T15:17:01.066478+0000","last_active":"2024-04-15T15:17:01.073110+0000","last_peered":"2024-04-15T15:17:01.073110+0000","last_clean":"2024-04-15T15:17:01.073110+0000","last_became_active":"2024-04-15T15:17:01.066229+0000","last_became_peered":"2024-04-15T15:17:01.066229+0000","last_unstale":"2024-04-15T15:17:01.073110+0000","last_undegraded":"2024-04-15T15:17:01.073110+0000","last_fullsized":"2024-04-15T15:17:01.073110+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.066213+0000","last_change":"2024-04-15T15:17:00.064123+0000","last_active":"2024-04-15T15:17:00.066213+0000","last_peered":"2024-04-15T15:17:00.066213+0000","last_clean":"2024-04-15T15:17:00.066213+0000","last_became_active":"2024-04-15T15:17:00.063535+0000","last_became_peered":"2024-04-15T15:17:00.063535+0000","last_unstale":"2024-04-15T15:17:00.066213+0000","last_undegraded":"2024-04-15T15:17:00.066213+0000","last_fullsized":"2024-04-15T15:17:00.066213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046719+0000","last_change":"2024-04-15T15:17:00.046719+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046719+0000","last_undegraded":"2024-04-15T15:17:00.046719+0000","last_fullsized":"2024-04-15T15:17:00.046719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053705+0000","last_change":"2024-04-15T15:17:00.051113+0000","last_active":"2024-04-15T15:17:00.053705+0000","last_peered":"2024-04-15T15:17:00.053705+0000","last_clean":"2024-04-15T15:17:00.053705+0000","last_became_active":"2024-04-15T15:17:00.050824+0000","last_became_peered":"2024-04-15T15:17:00.050824+0000","last_unstale":"2024-04-15T15:17:00.053705+0000","last_undegraded":"2024-04-15T15:17:00.053705+0000","last_fullsized":"2024-04-15T15:17:00.053705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:50:01.167747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043001+0000","last_change":"2024-04-15T15:17:00.043001+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043001+0000","last_undegraded":"2024-04-15T15:17:00.043001+0000","last_fullsized":"2024-04-15T15:17:00.043001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078424+0000","last_change":"2024-04-15T15:17:00.058333+0000","last_active":"2024-04-15T15:17:01.078424+0000","last_peered":"2024-04-15T15:17:01.078424+0000","last_clean":"2024-04-15T15:17:01.078424+0000","last_became_active":"2024-04-15T15:17:00.057892+0000","last_became_peered":"2024-04-15T15:17:00.057892+0000","last_unstale":"2024-04-15T15:17:01.078424+0000","last_undegraded":"2024-04-15T15:17:01.078424+0000","last_fullsized":"2024-04-15T15:17:01.078424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052245+0000","last_change":"2024-04-15T15:17:00.052245+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052245+0000","last_undegraded":"2024-04-15T15:17:00.052245+0000","last_fullsized":"2024-04-15T15:17:00.052245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053028+0000","last_change":"2024-04-15T15:17:00.049576+0000","last_active":"2024-04-15T15:17:00.053028+0000","last_peered":"2024-04-15T15:17:00.053028+0000","last_clean":"2024-04-15T15:17:00.053028+0000","last_became_active":"2024-04-15T15:17:00.049280+0000","last_became_peered":"2024-04-15T15:17:00.049280+0000","last_unstale":"2024-04-15T15:17:00.053028+0000","last_undegraded":"2024-04-15T15:17:00.053028+0000","last_fullsized":"2024-04-15T15:17:00.053028+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:16:00.136473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046328+0000","last_change":"2024-04-15T15:17:00.046328+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046328+0000","last_undegraded":"2024-04-15T15:17:00.046328+0000","last_fullsized":"2024-04-15T15:17:00.046328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.062211+0000","last_change":"2024-04-15T15:17:00.061061+0000","last_active":"2024-04-15T15:17:00.062211+0000","last_peered":"2024-04-15T15:17:00.062211+0000","last_clean":"2024-04-15T15:17:00.062211+0000","last_became_active":"2024-04-15T15:17:00.060731+0000","last_became_peered":"2024-04-15T15:17:00.060731+0000","last_unstale":"2024-04-15T15:17:00.062211+0000","last_undegraded":"2024-04-15T15:17:00.062211+0000","last_fullsized":"2024-04-15T15:17:00.062211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046258+0000","last_change":"2024-04-15T15:17:00.046258+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046258+0000","last_undegraded":"2024-04-15T15:17:00.046258+0000","last_fullsized":"2024-04-15T15:17:00.046258+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064288+0000","last_change":"2024-04-15T15:17:00.063764+0000","last_active":"2024-04-15T15:17:00.064288+0000","last_peered":"2024-04-15T15:17:00.064288+0000","last_clean":"2024-04-15T15:17:00.064288+0000","last_became_active":"2024-04-15T15:17:00.063528+0000","last_became_peered":"2024-04-15T15:17:00.063528+0000","last_unstale":"2024-04-15T15:17:00.064288+0000","last_undegraded":"2024-04-15T15:17:00.064288+0000","last_fullsized":"2024-04-15T15:17:00.064288+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043848+0000","last_change":"2024-04-15T15:17:00.043848+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043848+0000","last_undegraded":"2024-04-15T15:17:00.043848+0000","last_fullsized":"2024-04-15T15:17:00.043848+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048642+0000","last_change":"2024-04-15T15:17:00.056627+0000","last_active":"2024-04-15T15:17:01.048642+0000","last_peered":"2024-04-15T15:17:01.048642+0000","last_clean":"2024-04-15T15:17:01.048642+0000","last_became_active":"2024-04-15T15:17:00.056317+0000","last_became_peered":"2024-04-15T15:17:00.056317+0000","last_unstale":"2024-04-15T15:17:01.048642+0000","last_undegraded":"2024-04-15T15:17:01.048642+0000","last_fullsized":"2024-04-15T15:17:01.048642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074263+0000","last_change":"2024-04-15T15:17:01.071745+0000","last_active":"2024-04-15T15:17:01.074263+0000","last_peered":"2024-04-15T15:17:01.074263+0000","last_clean":"2024-04-15T15:17:01.074263+0000","last_became_active":"2024-04-15T15:17:01.071466+0000","last_became_peered":"2024-04-15T15:17:01.071466+0000","last_unstale":"2024-04-15T15:17:01.074263+0000","last_undegraded":"2024-04-15T15:17:01.074263+0000","last_fullsized":"2024-04-15T15:17:01.074263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:04.108334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.054345+0000","last_change":"2024-04-15T15:17:00.057863+0000","last_active":"2024-04-15T15:17:01.054345+0000","last_peered":"2024-04-15T15:17:01.054345+0000","last_clean":"2024-04-15T15:17:01.054345+0000","last_became_active":"2024-04-15T15:17:00.057596+0000","last_became_peered":"2024-04-15T15:17:00.057596+0000","last_unstale":"2024-04-15T15:17:01.054345+0000","last_undegraded":"2024-04-15T15:17:01.054345+0000","last_fullsized":"2024-04-15T15:17:01.054345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047469+0000","last_change":"2024-04-15T15:17:00.047469+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047469+0000","last_undegraded":"2024-04-15T15:17:00.047469+0000","last_fullsized":"2024-04-15T15:17:00.047469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.060185+0000","last_change":"2024-04-15T15:17:00.058811+0000","last_active":"2024-04-15T15:17:00.060185+0000","last_peered":"2024-04-15T15:17:00.060185+0000","last_clean":"2024-04-15T15:17:00.060185+0000","last_became_active":"2024-04-15T15:17:00.058530+0000","last_became_peered":"2024-04-15T15:17:00.058530+0000","last_unstale":"2024-04-15T15:17:00.060185+0000","last_undegraded":"2024-04-15T15:17:00.060185+0000","last_fullsized":"2024-04-15T15:17:00.060185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:03.103510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074626+0000","last_change":"2024-04-15T15:17:01.070460+0000","last_active":"2024-04-15T15:17:01.074626+0000","last_peered":"2024-04-15T15:17:01.074626+0000","last_clean":"2024-04-15T15:17:01.074626+0000","last_became_active":"2024-04-15T15:17:01.070110+0000","last_became_peered":"2024-04-15T15:17:01.070110+0000","last_unstale":"2024-04-15T15:17:01.074626+0000","last_undegraded":"2024-04-15T15:17:01.074626+0000","last_fullsized":"2024-04-15T15:17:01.074626+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064075+0000","last_change":"2024-04-15T15:17:00.061284+0000","last_active":"2024-04-15T15:17:00.064075+0000","last_peered":"2024-04-15T15:17:00.064075+0000","last_clean":"2024-04-15T15:17:00.064075+0000","last_became_active":"2024-04-15T15:17:00.060520+0000","last_became_peered":"2024-04-15T15:17:00.060520+0000","last_unstale":"2024-04-15T15:17:00.064075+0000","last_undegraded":"2024-04-15T15:17:00.064075+0000","last_fullsized":"2024-04-15T15:17:00.064075+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053524+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:00.053524+0000","last_peered":"2024-04-15T15:17:00.053524+0000","last_clean":"2024-04-15T15:17:00.053524+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:00.053524+0000","last_undegraded":"2024-04-15T15:17:00.053524+0000","last_fullsized":"2024-04-15T15:17:00.053524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":187741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1015808,"data_stored":770360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":179992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":173325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":179992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":763693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:05.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.235+0000 7f4c4adb5640 1 Processor -- start 2024-04-15T15:17:05.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c4adb5640 1 -- start start 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c4adb5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4c440d3270 0x7f4c440cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c4adb5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c440b47a0 con 0x7f4c440d2360 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c4adb5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c440b4a80 con 0x7f4c4415e700 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c4adb5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f4c440b53b0 con 0x7f4c440d3270 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c43fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4c440d2360 0x7f4c440d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52244/0 (socket says 172.21.15.46:52244) 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.236+0000 7f4c43fff640 1 -- 172.21.15.46:0/2349840179 learned_addr learned my addr 172.21.15.46:0/2349840179 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c49352640 1 --2- 172.21.15.46:0/2349840179 >> v2:172.21.15.78:3300/0 conn(0x7f4c440d3270 0x7f4c440cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c437fe640 1 -- 172.21.15.46:0/2349840179 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1408648761 0 0) 0x7f4c440b47a0 con 0x7f4c440d2360 2024-04-15T15:17:05.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c437fe640 1 -- 172.21.15.46:0/2349840179 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c34003520 con 0x7f4c440d2360 2024-04-15T15:17:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c49352640 1 -- 172.21.15.46:0/2349840179 >> v1:172.21.15.78:6789/0 conn(0x7f4c4415e700 legacy=0x7f4c440b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c49352640 1 -- 172.21.15.46:0/2349840179 >> v1:172.21.15.46:6789/0 conn(0x7f4c440d2360 legacy=0x7f4c440d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c437fe640 1 -- 172.21.15.46:0/2349840179 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 100093496 0 0) 0x7f4c34003520 con 0x7f4c440d2360 2024-04-15T15:17:05.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.237+0000 7f4c49352640 1 -- 172.21.15.46:0/2349840179 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c440b4d60 con 0x7f4c440d3270 2024-04-15T15:17:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.238+0000 7f4c49352640 1 --2- 172.21.15.46:0/2349840179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cf370 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c440b5530 tx=0x7f4c3c02be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d662a189f12fbc49 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.238+0000 7f4c437fe640 1 -- 172.21.15.46:0/2349840179 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c039070 con 0x7f4c440d3270 2024-04-15T15:17:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.238+0000 7f4c437fe640 1 -- 172.21.15.46:0/2349840179 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c3c02c930 con 0x7f4c440d3270 2024-04-15T15:17:05.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.238+0000 7f4c437fe640 1 -- 172.21.15.46:0/2349840179 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30001670 con 0x7f4c440d3270 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.238+0000 7f4c4adb5640 1 -- 172.21.15.46:0/2349840179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 msgr2=0x7f4c440cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.238+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/2349840179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cf370 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c440b5530 tx=0x7f4c3c02be00 comp rx=0 tx=0).stop 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.239+0000 7f4c4adb5640 1 -- 172.21.15.46:0/2349840179 shutdown_connections 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.239+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/2349840179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cf370 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.239+0000 7f4c4adb5640 1 -- 172.21.15.46:0/2349840179 >> 172.21.15.46:0/2349840179 conn(0x7f4c4408bf60 msgr2=0x7f4c440d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.239+0000 7f4c4adb5640 1 -- 172.21.15.46:0/2349840179 shutdown_connections 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.239+0000 7f4c4adb5640 1 -- 172.21.15.46:0/2349840179 wait complete. 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.240+0000 7f4c4adb5640 1 Processor -- start 2024-04-15T15:17:05.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.240+0000 7f4c4adb5640 1 -- start start 2024-04-15T15:17:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 0x7f4c440caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c4415e700 0x7f4c4415eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c440d1d40 con 0x7f4c440d2360 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4c440b4450 con 0x7f4c440d3270 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4c440d1ec0 con 0x7f4c4415e700 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 0x7f4c440caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 0x7f4c440caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48168/0 (socket says 172.21.15.46:48168) 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 -- 172.21.15.46:0/3136212494 learned_addr learned my addr 172.21.15.46:0/3136212494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c49352640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c4415e700 0x7f4c4415eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c43fff640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 -- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c4415e700 msgr2=0x7f4c4415eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c4415e700 0x7f4c4415eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 -- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 msgr2=0x7f4c440cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 -- 172.21.15.46:0/3136212494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c440cd010 con 0x7f4c440d2360 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c43fff640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c48b51640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 0x7f4c440caa90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800c3c0 tx=0x7f4c3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c380052d0 con 0x7f4c440d2360 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c380054b0 con 0x7f4c440d2360 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c440cd2f0 con 0x7f4c440d2360 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.241+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38006a40 con 0x7f4c440d2360 2024-04-15T15:17:05.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.242+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c440cd8b0 con 0x7f4c440d2360 2024-04-15T15:17:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.242+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c440689e0 con 0x7f4c440d2360 2024-04-15T15:17:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.243+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c38006be0 con 0x7f4c440d2360 2024-04-15T15:17:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.243+0000 7f4c417fa640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c2806c8f0 0x7f4c2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.243+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4c3808b1d0 con 0x7f4c440d2360 2024-04-15T15:17:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.244+0000 7f4c43fff640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c2806c8f0 0x7f4c2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.244+0000 7f4c43fff640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c2806c8f0 0x7f4c2806edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4c300011e0 tx=0x7f4c30009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.248+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c380593d0 con 0x7f4c440d2360 2024-04-15T15:17:05.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.428+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4c440d2d70 con 0x7f4c2806c8f0 2024-04-15T15:17:05.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.434+0000 7f4c417fa640 1 -- 172.21.15.46:0/3136212494 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248614 (secure 0 0 0) 0x7f4c440d2d70 con 0x7f4c2806c8f0 2024-04-15T15:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:05.441 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:05.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c2806c8f0 msgr2=0x7f4c2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c2806c8f0 0x7f4c2806edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4c300011e0 tx=0x7f4c30009040 comp rx=0 tx=0).stop 2024-04-15T15:17:05.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 msgr2=0x7f4c440caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 0x7f4c440caa90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4c3800c3c0 tx=0x7f4c3800c880 comp rx=0 tx=0).stop 2024-04-15T15:17:05.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 shutdown_connections 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4c2806c8f0 0x7f4c2806edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4c4415e700 0x7f4c4415eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c440d3270 0x7f4c440cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 --2- 172.21.15.46:0/3136212494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c440d2360 0x7f4c440caa90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 >> 172.21.15.46:0/3136212494 conn(0x7f4c4408bf60 msgr2=0x7f4c440b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 shutdown_connections 2024-04-15T15:17:05.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.437+0000 7f4c4adb5640 1 -- 172.21.15.46:0/3136212494 wait complete. 2024-04-15T15:17:05.450 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-15T15:17:04.373266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237304,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737416,"statfs":{"total":773094113280,"available":772851113984,"internally_reserved":0,"allocated":4595712,"data_stored":2630137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.338094"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048962+0000","last_change":"2024-04-15T15:17:00.048962+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048962+0000","last_undegraded":"2024-04-15T15:17:00.048962+0000","last_fullsized":"2024-04-15T15:17:00.048962+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046889+0000","last_change":"2024-04-15T15:17:00.046889+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046889+0000","last_undegraded":"2024-04-15T15:17:00.046889+0000","last_fullsized":"2024-04-15T15:17:00.046889+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047885+0000","last_change":"2024-04-15T15:17:00.047885+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047885+0000","last_undegraded":"2024-04-15T15:17:00.047885+0000","last_fullsized":"2024-04-15T15:17:00.047885+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045994+0000","last_change":"2024-04-15T15:17:00.045994+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045994+0000","last_undegraded":"2024-04-15T15:17:00.045994+0000","last_fullsized":"2024-04-15T15:17:00.045994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043764+0000","last_change":"2024-04-15T15:17:00.043764+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043764+0000","last_undegraded":"2024-04-15T15:17:00.043764+0000","last_fullsized":"2024-04-15T15:17:00.043764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044975+0000","last_change":"2024-04-15T15:17:00.044975+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044975+0000","last_undegraded":"2024-04-15T15:17:00.044975+0000","last_fullsized":"2024-04-15T15:17:00.044975+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074681+0000","last_change":"2024-04-15T15:17:01.069002+0000","last_active":"2024-04-15T15:17:01.074681+0000","last_peered":"2024-04-15T15:17:01.074681+0000","last_clean":"2024-04-15T15:17:01.074681+0000","last_became_active":"2024-04-15T15:17:01.068755+0000","last_became_peered":"2024-04-15T15:17:01.068755+0000","last_unstale":"2024-04-15T15:17:01.074681+0000","last_undegraded":"2024-04-15T15:17:01.074681+0000","last_fullsized":"2024-04-15T15:17:01.074681+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.049417+0000","last_change":"2024-04-15T15:17:00.049417+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.049417+0000","last_undegraded":"2024-04-15T15:17:00.049417+0000","last_fullsized":"2024-04-15T15:17:00.049417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.049028+0000","last_change":"2024-04-15T15:17:00.049028+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.049028+0000","last_undegraded":"2024-04-15T15:17:00.049028+0000","last_fullsized":"2024-04-15T15:17:00.049028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073089+0000","last_change":"2024-04-15T15:17:01.068258+0000","last_active":"2024-04-15T15:17:01.073089+0000","last_peered":"2024-04-15T15:17:01.073089+0000","last_clean":"2024-04-15T15:17:01.073089+0000","last_became_active":"2024-04-15T15:17:01.067888+0000","last_became_peered":"2024-04-15T15:17:01.067888+0000","last_unstale":"2024-04-15T15:17:01.073089+0000","last_undegraded":"2024-04-15T15:17:01.073089+0000","last_fullsized":"2024-04-15T15:17:01.073089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075397+0000","last_change":"2024-04-15T15:17:01.070976+0000","last_active":"2024-04-15T15:17:01.075397+0000","last_peered":"2024-04-15T15:17:01.075397+0000","last_clean":"2024-04-15T15:17:01.075397+0000","last_became_active":"2024-04-15T15:17:01.070678+0000","last_became_peered":"2024-04-15T15:17:01.070678+0000","last_unstale":"2024-04-15T15:17:01.075397+0000","last_undegraded":"2024-04-15T15:17:01.075397+0000","last_fullsized":"2024-04-15T15:17:01.075397+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:38:18.560062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045186+0000","last_change":"2024-04-15T15:17:00.045186+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045186+0000","last_undegraded":"2024-04-15T15:17:00.045186+0000","last_fullsized":"2024-04-15T15:17:00.045186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.065827+0000","last_change":"2024-04-15T15:17:01.064962+0000","last_active":"2024-04-15T15:17:01.065827+0000","last_peered":"2024-04-15T15:17:01.065827+0000","last_clean":"2024-04-15T15:17:01.065827+0000","last_became_active":"2024-04-15T15:17:01.064395+0000","last_became_peered":"2024-04-15T15:17:01.064395+0000","last_unstale":"2024-04-15T15:17:01.065827+0000","last_undegraded":"2024-04-15T15:17:01.065827+0000","last_fullsized":"2024-04-15T15:17:01.065827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072393+0000","last_change":"2024-04-15T15:17:01.069059+0000","last_active":"2024-04-15T15:17:01.072393+0000","last_peered":"2024-04-15T15:17:01.072393+0000","last_clean":"2024-04-15T15:17:01.072393+0000","last_became_active":"2024-04-15T15:17:01.067835+0000","last_became_peered":"2024-04-15T15:17:01.067835+0000","last_unstale":"2024-04-15T15:17:01.072393+0000","last_undegraded":"2024-04-15T15:17:01.072393+0000","last_fullsized":"2024-04-15T15:17:01.072393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073052+0000","last_change":"2024-04-15T15:17:01.065939+0000","last_active":"2024-04-15T15:17:01.073052+0000","last_peered":"2024-04-15T15:17:01.073052+0000","last_clean":"2024-04-15T15:17:01.073052+0000","last_became_active":"2024-04-15T15:17:01.065641+0000","last_became_peered":"2024-04-15T15:17:01.065641+0000","last_unstale":"2024-04-15T15:17:01.073052+0000","last_undegraded":"2024-04-15T15:17:01.073052+0000","last_fullsized":"2024-04-15T15:17:01.073052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052205+0000","last_change":"2024-04-15T15:17:00.052205+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052205+0000","last_undegraded":"2024-04-15T15:17:00.052205+0000","last_fullsized":"2024-04-15T15:17:00.052205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047966+0000","last_change":"2024-04-15T15:17:00.047966+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047966+0000","last_undegraded":"2024-04-15T15:17:00.047966+0000","last_fullsized":"2024-04-15T15:17:00.047966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073211+0000","last_change":"2024-04-15T15:17:01.070470+0000","last_active":"2024-04-15T15:17:01.073211+0000","last_peered":"2024-04-15T15:17:01.073211+0000","last_clean":"2024-04-15T15:17:01.073211+0000","last_became_active":"2024-04-15T15:17:01.070200+0000","last_became_peered":"2024-04-15T15:17:01.070200+0000","last_unstale":"2024-04-15T15:17:01.073211+0000","last_undegraded":"2024-04-15T15:17:01.073211+0000","last_fullsized":"2024-04-15T15:17:01.073211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043158+0000","last_change":"2024-04-15T15:17:00.043158+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043158+0000","last_undegraded":"2024-04-15T15:17:00.043158+0000","last_fullsized":"2024-04-15T15:17:00.043158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.076558+0000","last_change":"2024-04-15T15:17:01.071687+0000","last_active":"2024-04-15T15:17:01.076558+0000","last_peered":"2024-04-15T15:17:01.076558+0000","last_clean":"2024-04-15T15:17:01.076558+0000","last_became_active":"2024-04-15T15:17:01.071330+0000","last_became_peered":"2024-04-15T15:17:01.071330+0000","last_unstale":"2024-04-15T15:17:01.076558+0000","last_undegraded":"2024-04-15T15:17:01.076558+0000","last_fullsized":"2024-04-15T15:17:01.076558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073107+0000","last_change":"2024-04-15T15:17:01.067759+0000","last_active":"2024-04-15T15:17:01.073107+0000","last_peered":"2024-04-15T15:17:01.073107+0000","last_clean":"2024-04-15T15:17:01.073107+0000","last_became_active":"2024-04-15T15:17:01.067476+0000","last_became_peered":"2024-04-15T15:17:01.067476+0000","last_unstale":"2024-04-15T15:17:01.073107+0000","last_undegraded":"2024-04-15T15:17:01.073107+0000","last_fullsized":"2024-04-15T15:17:01.073107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072415+0000","last_change":"2024-04-15T15:17:01.070298+0000","last_active":"2024-04-15T15:17:01.072415+0000","last_peered":"2024-04-15T15:17:01.072415+0000","last_clean":"2024-04-15T15:17:01.072415+0000","last_became_active":"2024-04-15T15:17:01.070032+0000","last_became_peered":"2024-04-15T15:17:01.070032+0000","last_unstale":"2024-04-15T15:17:01.072415+0000","last_undegraded":"2024-04-15T15:17:01.072415+0000","last_fullsized":"2024-04-15T15:17:01.072415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:23.034253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052838+0000","last_change":"2024-04-15T15:17:00.052838+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052838+0000","last_undegraded":"2024-04-15T15:17:00.052838+0000","last_fullsized":"2024-04-15T15:17:00.052838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.042458+0000","last_change":"2024-04-15T15:17:00.042458+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.042458+0000","last_undegraded":"2024-04-15T15:17:00.042458+0000","last_fullsized":"2024-04-15T15:17:00.042458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061210+0000","last_change":"2024-04-15T15:17:01.056648+0000","last_active":"2024-04-15T15:17:01.061210+0000","last_peered":"2024-04-15T15:17:01.061210+0000","last_clean":"2024-04-15T15:17:01.061210+0000","last_became_active":"2024-04-15T15:17:01.056354+0000","last_became_peered":"2024-04-15T15:17:01.056354+0000","last_unstale":"2024-04-15T15:17:01.061210+0000","last_undegraded":"2024-04-15T15:17:01.061210+0000","last_fullsized":"2024-04-15T15:17:01.061210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044860+0000","last_change":"2024-04-15T15:17:00.044860+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044860+0000","last_undegraded":"2024-04-15T15:17:00.044860+0000","last_fullsized":"2024-04-15T15:17:00.044860+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.062082+0000","last_change":"2024-04-15T15:17:01.059015+0000","last_active":"2024-04-15T15:17:01.062082+0000","last_peered":"2024-04-15T15:17:01.062082+0000","last_clean":"2024-04-15T15:17:01.062082+0000","last_became_active":"2024-04-15T15:17:01.058718+0000","last_became_peered":"2024-04-15T15:17:01.058718+0000","last_unstale":"2024-04-15T15:17:01.062082+0000","last_undegraded":"2024-04-15T15:17:01.062082+0000","last_fullsized":"2024-04-15T15:17:01.062082+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074673+0000","last_change":"2024-04-15T15:17:01.069694+0000","last_active":"2024-04-15T15:17:01.074673+0000","last_peered":"2024-04-15T15:17:01.074673+0000","last_clean":"2024-04-15T15:17:01.074673+0000","last_became_active":"2024-04-15T15:17:01.069408+0000","last_became_peered":"2024-04-15T15:17:01.069408+0000","last_unstale":"2024-04-15T15:17:01.074673+0000","last_undegraded":"2024-04-15T15:17:01.074673+0000","last_fullsized":"2024-04-15T15:17:01.074673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072620+0000","last_change":"2024-04-15T15:17:01.068919+0000","last_active":"2024-04-15T15:17:01.072620+0000","last_peered":"2024-04-15T15:17:01.072620+0000","last_clean":"2024-04-15T15:17:01.072620+0000","last_became_active":"2024-04-15T15:17:01.068639+0000","last_became_peered":"2024-04-15T15:17:01.068639+0000","last_unstale":"2024-04-15T15:17:01.072620+0000","last_undegraded":"2024-04-15T15:17:01.072620+0000","last_fullsized":"2024-04-15T15:17:01.072620+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061418+0000","last_change":"2024-04-15T15:17:01.056050+0000","last_active":"2024-04-15T15:17:01.061418+0000","last_peered":"2024-04-15T15:17:01.061418+0000","last_clean":"2024-04-15T15:17:01.061418+0000","last_became_active":"2024-04-15T15:17:01.055735+0000","last_became_peered":"2024-04-15T15:17:01.055735+0000","last_unstale":"2024-04-15T15:17:01.061418+0000","last_undegraded":"2024-04-15T15:17:01.061418+0000","last_fullsized":"2024-04-15T15:17:01.061418+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:51.299505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048125+0000","last_change":"2024-04-15T15:17:00.048125+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048125+0000","last_undegraded":"2024-04-15T15:17:00.048125+0000","last_fullsized":"2024-04-15T15:17:00.048125+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044106+0000","last_change":"2024-04-15T15:17:00.044106+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044106+0000","last_undegraded":"2024-04-15T15:17:00.044106+0000","last_fullsized":"2024-04-15T15:17:00.044106+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047283+0000","last_change":"2024-04-15T15:17:00.047283+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047283+0000","last_undegraded":"2024-04-15T15:17:00.047283+0000","last_fullsized":"2024-04-15T15:17:00.047283+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078586+0000","last_change":"2024-04-15T15:17:00.053485+0000","last_active":"2024-04-15T15:17:01.078586+0000","last_peered":"2024-04-15T15:17:01.078586+0000","last_clean":"2024-04-15T15:17:01.078586+0000","last_became_active":"2024-04-15T15:17:00.053211+0000","last_became_peered":"2024-04-15T15:17:00.053211+0000","last_unstale":"2024-04-15T15:17:01.078586+0000","last_undegraded":"2024-04-15T15:17:01.078586+0000","last_fullsized":"2024-04-15T15:17:01.078586+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043633+0000","last_change":"2024-04-15T15:17:00.043633+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043633+0000","last_undegraded":"2024-04-15T15:17:00.043633+0000","last_fullsized":"2024-04-15T15:17:00.043633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058716+0000","last_change":"2024-04-15T15:17:00.056209+0000","last_active":"2024-04-15T15:17:00.058716+0000","last_peered":"2024-04-15T15:17:00.058716+0000","last_clean":"2024-04-15T15:17:00.058716+0000","last_became_active":"2024-04-15T15:17:00.055881+0000","last_became_peered":"2024-04-15T15:17:00.055881+0000","last_unstale":"2024-04-15T15:17:00.058716+0000","last_undegraded":"2024-04-15T15:17:00.058716+0000","last_fullsized":"2024-04-15T15:17:00.058716+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072065+0000","last_change":"2024-04-15T15:17:01.067226+0000","last_active":"2024-04-15T15:17:01.072065+0000","last_peered":"2024-04-15T15:17:01.072065+0000","last_clean":"2024-04-15T15:17:01.072065+0000","last_became_active":"2024-04-15T15:17:01.066479+0000","last_became_peered":"2024-04-15T15:17:01.066479+0000","last_unstale":"2024-04-15T15:17:01.072065+0000","last_undegraded":"2024-04-15T15:17:01.072065+0000","last_fullsized":"2024-04-15T15:17:01.072065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058483+0000","last_change":"2024-04-15T15:17:00.054251+0000","last_active":"2024-04-15T15:17:00.058483+0000","last_peered":"2024-04-15T15:17:00.058483+0000","last_clean":"2024-04-15T15:17:00.058483+0000","last_became_active":"2024-04-15T15:17:00.053980+0000","last_became_peered":"2024-04-15T15:17:00.053980+0000","last_unstale":"2024-04-15T15:17:00.058483+0000","last_undegraded":"2024-04-15T15:17:00.058483+0000","last_fullsized":"2024-04-15T15:17:00.058483+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.050290+0000","last_change":"2024-04-15T15:17:00.050290+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.050290+0000","last_undegraded":"2024-04-15T15:17:00.050290+0000","last_fullsized":"2024-04-15T15:17:00.050290+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.080694+0000","last_change":"2024-04-15T15:17:00.058378+0000","last_active":"2024-04-15T15:17:01.080694+0000","last_peered":"2024-04-15T15:17:01.080694+0000","last_clean":"2024-04-15T15:17:01.080694+0000","last_became_active":"2024-04-15T15:17:00.058024+0000","last_became_peered":"2024-04-15T15:17:00.058024+0000","last_unstale":"2024-04-15T15:17:01.080694+0000","last_undegraded":"2024-04-15T15:17:01.080694+0000","last_fullsized":"2024-04-15T15:17:01.080694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047108+0000","last_change":"2024-04-15T15:17:00.047108+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047108+0000","last_undegraded":"2024-04-15T15:17:00.047108+0000","last_fullsized":"2024-04-15T15:17:00.047108+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.077676+0000","last_change":"2024-04-15T15:17:00.052588+0000","last_active":"2024-04-15T15:17:01.077676+0000","last_peered":"2024-04-15T15:17:01.077676+0000","last_clean":"2024-04-15T15:17:01.077676+0000","last_became_active":"2024-04-15T15:17:00.052287+0000","last_became_peered":"2024-04-15T15:17:00.052287+0000","last_unstale":"2024-04-15T15:17:01.077676+0000","last_undegraded":"2024-04-15T15:17:01.077676+0000","last_fullsized":"2024-04-15T15:17:01.077676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079604+0000","last_change":"2024-04-15T15:17:01.077529+0000","last_active":"2024-04-15T15:17:01.079604+0000","last_peered":"2024-04-15T15:17:01.079604+0000","last_clean":"2024-04-15T15:17:01.079604+0000","last_became_active":"2024-04-15T15:17:01.077231+0000","last_became_peered":"2024-04-15T15:17:01.077231+0000","last_unstale":"2024-04-15T15:17:01.079604+0000","last_undegraded":"2024-04-15T15:17:01.079604+0000","last_fullsized":"2024-04-15T15:17:01.079604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:58:51.072457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.063827+0000","last_change":"2024-04-15T15:17:00.060924+0000","last_active":"2024-04-15T15:17:00.063827+0000","last_peered":"2024-04-15T15:17:00.063827+0000","last_clean":"2024-04-15T15:17:00.063827+0000","last_became_active":"2024-04-15T15:17:00.060610+0000","last_became_peered":"2024-04-15T15:17:00.060610+0000","last_unstale":"2024-04-15T15:17:00.063827+0000","last_undegraded":"2024-04-15T15:17:00.063827+0000","last_fullsized":"2024-04-15T15:17:00.063827+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072370+0000","last_change":"2024-04-15T15:17:01.068809+0000","last_active":"2024-04-15T15:17:01.072370+0000","last_peered":"2024-04-15T15:17:01.072370+0000","last_clean":"2024-04-15T15:17:01.072370+0000","last_became_active":"2024-04-15T15:17:01.067740+0000","last_became_peered":"2024-04-15T15:17:01.067740+0000","last_unstale":"2024-04-15T15:17:01.072370+0000","last_undegraded":"2024-04-15T15:17:01.072370+0000","last_fullsized":"2024-04-15T15:17:01.072370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058130+0000","last_change":"2024-04-15T15:17:00.056079+0000","last_active":"2024-04-15T15:17:00.058130+0000","last_peered":"2024-04-15T15:17:00.058130+0000","last_clean":"2024-04-15T15:17:00.058130+0000","last_became_active":"2024-04-15T15:17:00.055794+0000","last_became_peered":"2024-04-15T15:17:00.055794+0000","last_unstale":"2024-04-15T15:17:00.058130+0000","last_undegraded":"2024-04-15T15:17:00.058130+0000","last_fullsized":"2024-04-15T15:17:00.058130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.050788+0000","last_change":"2024-04-15T15:17:00.050788+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.050788+0000","last_undegraded":"2024-04-15T15:17:00.050788+0000","last_fullsized":"2024-04-15T15:17:00.050788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056679+0000","last_change":"2024-04-15T15:17:00.048501+0000","last_active":"2024-04-15T15:17:01.056679+0000","last_peered":"2024-04-15T15:17:01.056679+0000","last_clean":"2024-04-15T15:17:01.056679+0000","last_became_active":"2024-04-15T15:17:00.048148+0000","last_became_peered":"2024-04-15T15:17:00.048148+0000","last_unstale":"2024-04-15T15:17:01.056679+0000","last_undegraded":"2024-04-15T15:17:01.056679+0000","last_fullsized":"2024-04-15T15:17:01.056679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043571+0000","last_change":"2024-04-15T15:17:00.043571+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043571+0000","last_undegraded":"2024-04-15T15:17:00.043571+0000","last_fullsized":"2024-04-15T15:17:00.043571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.060211+0000","last_change":"2024-04-15T15:17:00.059059+0000","last_active":"2024-04-15T15:17:00.060211+0000","last_peered":"2024-04-15T15:17:00.060211+0000","last_clean":"2024-04-15T15:17:00.060211+0000","last_became_active":"2024-04-15T15:17:00.058527+0000","last_became_peered":"2024-04-15T15:17:00.058527+0000","last_unstale":"2024-04-15T15:17:00.060211+0000","last_undegraded":"2024-04-15T15:17:00.060211+0000","last_fullsized":"2024-04-15T15:17:00.060211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052662+0000","last_change":"2024-04-15T15:17:00.052662+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052662+0000","last_undegraded":"2024-04-15T15:17:00.052662+0000","last_fullsized":"2024-04-15T15:17:00.052662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056094+0000","last_change":"2024-04-15T15:17:00.058594+0000","last_active":"2024-04-15T15:17:01.056094+0000","last_peered":"2024-04-15T15:17:01.056094+0000","last_clean":"2024-04-15T15:17:01.056094+0000","last_became_active":"2024-04-15T15:17:00.058359+0000","last_became_peered":"2024-04-15T15:17:00.058359+0000","last_unstale":"2024-04-15T15:17:01.056094+0000","last_undegraded":"2024-04-15T15:17:01.056094+0000","last_fullsized":"2024-04-15T15:17:01.056094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045457+0000","last_change":"2024-04-15T15:17:00.045457+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045457+0000","last_undegraded":"2024-04-15T15:17:00.045457+0000","last_fullsized":"2024-04-15T15:17:00.045457+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078903+0000","last_change":"2024-04-15T15:17:00.055108+0000","last_active":"2024-04-15T15:17:01.078903+0000","last_peered":"2024-04-15T15:17:01.078903+0000","last_clean":"2024-04-15T15:17:01.078903+0000","last_became_active":"2024-04-15T15:17:00.054818+0000","last_became_peered":"2024-04-15T15:17:00.054818+0000","last_unstale":"2024-04-15T15:17:01.078903+0000","last_undegraded":"2024-04-15T15:17:01.078903+0000","last_fullsized":"2024-04-15T15:17:01.078903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072041+0000","last_change":"2024-04-15T15:17:01.066511+0000","last_active":"2024-04-15T15:17:01.072041+0000","last_peered":"2024-04-15T15:17:01.072041+0000","last_clean":"2024-04-15T15:17:01.072041+0000","last_became_active":"2024-04-15T15:17:01.066201+0000","last_became_peered":"2024-04-15T15:17:01.066201+0000","last_unstale":"2024-04-15T15:17:01.072041+0000","last_undegraded":"2024-04-15T15:17:01.072041+0000","last_fullsized":"2024-04-15T15:17:01.072041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.062191+0000","last_change":"2024-04-15T15:17:00.061109+0000","last_active":"2024-04-15T15:17:00.062191+0000","last_peered":"2024-04-15T15:17:00.062191+0000","last_clean":"2024-04-15T15:17:00.062191+0000","last_became_active":"2024-04-15T15:17:00.060823+0000","last_became_peered":"2024-04-15T15:17:00.060823+0000","last_unstale":"2024-04-15T15:17:00.062191+0000","last_undegraded":"2024-04-15T15:17:00.062191+0000","last_fullsized":"2024-04-15T15:17:00.062191+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046672+0000","last_change":"2024-04-15T15:17:00.046672+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046672+0000","last_undegraded":"2024-04-15T15:17:00.046672+0000","last_fullsized":"2024-04-15T15:17:00.046672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058974+0000","last_change":"2024-04-15T15:17:00.057661+0000","last_active":"2024-04-15T15:17:00.058974+0000","last_peered":"2024-04-15T15:17:00.058974+0000","last_clean":"2024-04-15T15:17:00.058974+0000","last_became_active":"2024-04-15T15:17:00.057376+0000","last_became_peered":"2024-04-15T15:17:00.057376+0000","last_unstale":"2024-04-15T15:17:00.058974+0000","last_undegraded":"2024-04-15T15:17:00.058974+0000","last_fullsized":"2024-04-15T15:17:00.058974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:22.029429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048446+0000","last_change":"2024-04-15T15:17:00.048446+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048446+0000","last_undegraded":"2024-04-15T15:17:00.048446+0000","last_fullsized":"2024-04-15T15:17:00.048446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058358+0000","last_change":"2024-04-15T15:17:00.055785+0000","last_active":"2024-04-15T15:17:00.058358+0000","last_peered":"2024-04-15T15:17:00.058358+0000","last_clean":"2024-04-15T15:17:00.058358+0000","last_became_active":"2024-04-15T15:17:00.055509+0000","last_became_peered":"2024-04-15T15:17:00.055509+0000","last_unstale":"2024-04-15T15:17:00.058358+0000","last_undegraded":"2024-04-15T15:17:00.058358+0000","last_fullsized":"2024-04-15T15:17:00.058358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047149+0000","last_change":"2024-04-15T15:17:00.047149+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047149+0000","last_undegraded":"2024-04-15T15:17:00.047149+0000","last_fullsized":"2024-04-15T15:17:00.047149+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048522+0000","last_change":"2024-04-15T15:17:00.055842+0000","last_active":"2024-04-15T15:17:01.048522+0000","last_peered":"2024-04-15T15:17:01.048522+0000","last_clean":"2024-04-15T15:17:01.048522+0000","last_became_active":"2024-04-15T15:17:00.054053+0000","last_became_peered":"2024-04-15T15:17:00.054053+0000","last_unstale":"2024-04-15T15:17:01.048522+0000","last_undegraded":"2024-04-15T15:17:01.048522+0000","last_fullsized":"2024-04-15T15:17:01.048522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047661+0000","last_change":"2024-04-15T15:17:00.047661+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047661+0000","last_undegraded":"2024-04-15T15:17:00.047661+0000","last_fullsized":"2024-04-15T15:17:00.047661+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064003+0000","last_change":"2024-04-15T15:17:00.061010+0000","last_active":"2024-04-15T15:17:00.064003+0000","last_peered":"2024-04-15T15:17:00.064003+0000","last_clean":"2024-04-15T15:17:00.064003+0000","last_became_active":"2024-04-15T15:17:00.060723+0000","last_became_peered":"2024-04-15T15:17:00.060723+0000","last_unstale":"2024-04-15T15:17:00.064003+0000","last_undegraded":"2024-04-15T15:17:00.064003+0000","last_fullsized":"2024-04-15T15:17:00.064003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.051279+0000","last_change":"2024-04-15T15:17:00.051279+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.051279+0000","last_undegraded":"2024-04-15T15:17:00.051279+0000","last_fullsized":"2024-04-15T15:17:00.051279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078087+0000","last_change":"2024-04-15T15:17:00.053088+0000","last_active":"2024-04-15T15:17:01.078087+0000","last_peered":"2024-04-15T15:17:01.078087+0000","last_clean":"2024-04-15T15:17:01.078087+0000","last_became_active":"2024-04-15T15:17:00.052762+0000","last_became_peered":"2024-04-15T15:17:00.052762+0000","last_unstale":"2024-04-15T15:17:01.078087+0000","last_undegraded":"2024-04-15T15:17:01.078087+0000","last_fullsized":"2024-04-15T15:17:01.078087+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075375+0000","last_change":"2024-04-15T15:17:01.067435+0000","last_active":"2024-04-15T15:17:01.075375+0000","last_peered":"2024-04-15T15:17:01.075375+0000","last_clean":"2024-04-15T15:17:01.075375+0000","last_became_active":"2024-04-15T15:17:01.066970+0000","last_became_peered":"2024-04-15T15:17:01.066970+0000","last_unstale":"2024-04-15T15:17:01.075375+0000","last_undegraded":"2024-04-15T15:17:01.075375+0000","last_fullsized":"2024-04-15T15:17:01.075375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079013+0000","last_change":"2024-04-15T15:17:00.049730+0000","last_active":"2024-04-15T15:17:01.079013+0000","last_peered":"2024-04-15T15:17:01.079013+0000","last_clean":"2024-04-15T15:17:01.079013+0000","last_became_active":"2024-04-15T15:17:00.049472+0000","last_became_peered":"2024-04-15T15:17:00.049472+0000","last_unstale":"2024-04-15T15:17:01.079013+0000","last_undegraded":"2024-04-15T15:17:01.079013+0000","last_fullsized":"2024-04-15T15:17:01.079013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072086+0000","last_change":"2024-04-15T15:17:01.067474+0000","last_active":"2024-04-15T15:17:01.072086+0000","last_peered":"2024-04-15T15:17:01.072086+0000","last_clean":"2024-04-15T15:17:01.072086+0000","last_became_active":"2024-04-15T15:17:01.066802+0000","last_became_peered":"2024-04-15T15:17:01.066802+0000","last_unstale":"2024-04-15T15:17:01.072086+0000","last_undegraded":"2024-04-15T15:17:01.072086+0000","last_fullsized":"2024-04-15T15:17:01.072086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.055389+0000","last_change":"2024-04-15T15:17:00.054252+0000","last_active":"2024-04-15T15:17:00.055389+0000","last_peered":"2024-04-15T15:17:00.055389+0000","last_clean":"2024-04-15T15:17:00.055389+0000","last_became_active":"2024-04-15T15:17:00.053917+0000","last_became_peered":"2024-04-15T15:17:00.053917+0000","last_unstale":"2024-04-15T15:17:00.055389+0000","last_undegraded":"2024-04-15T15:17:00.055389+0000","last_fullsized":"2024-04-15T15:17:00.055389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:53:53.053021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043001+0000","last_change":"2024-04-15T15:17:00.043001+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043001+0000","last_undegraded":"2024-04-15T15:17:00.043001+0000","last_fullsized":"2024-04-15T15:17:00.043001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.057998+0000","last_change":"2024-04-15T15:17:00.053573+0000","last_active":"2024-04-15T15:17:00.057998+0000","last_peered":"2024-04-15T15:17:00.057998+0000","last_clean":"2024-04-15T15:17:00.057998+0000","last_became_active":"2024-04-15T15:17:00.053296+0000","last_became_peered":"2024-04-15T15:17:00.053296+0000","last_unstale":"2024-04-15T15:17:00.057998+0000","last_undegraded":"2024-04-15T15:17:00.057998+0000","last_fullsized":"2024-04-15T15:17:00.057998+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047704+0000","last_change":"2024-04-15T15:17:00.047704+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047704+0000","last_undegraded":"2024-04-15T15:17:00.047704+0000","last_fullsized":"2024-04-15T15:17:00.047704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.055352+0000","last_change":"2024-04-15T15:17:00.052685+0000","last_active":"2024-04-15T15:17:00.055352+0000","last_peered":"2024-04-15T15:17:00.055352+0000","last_clean":"2024-04-15T15:17:00.055352+0000","last_became_active":"2024-04-15T15:17:00.052398+0000","last_became_peered":"2024-04-15T15:17:00.052398+0000","last_unstale":"2024-04-15T15:17:00.055352+0000","last_undegraded":"2024-04-15T15:17:00.055352+0000","last_fullsized":"2024-04-15T15:17:00.055352+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:42:40.378774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048032+0000","last_change":"2024-04-15T15:17:00.048032+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048032+0000","last_undegraded":"2024-04-15T15:17:00.048032+0000","last_fullsized":"2024-04-15T15:17:00.048032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.046519+0000","last_change":"2024-04-15T15:17:00.044538+0000","last_active":"2024-04-15T15:17:00.046519+0000","last_peered":"2024-04-15T15:17:00.046519+0000","last_clean":"2024-04-15T15:17:00.046519+0000","last_became_active":"2024-04-15T15:17:00.044237+0000","last_became_peered":"2024-04-15T15:17:00.044237+0000","last_unstale":"2024-04-15T15:17:00.046519+0000","last_undegraded":"2024-04-15T15:17:00.046519+0000","last_fullsized":"2024-04-15T15:17:00.046519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:42.468987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073110+0000","last_change":"2024-04-15T15:17:01.066478+0000","last_active":"2024-04-15T15:17:01.073110+0000","last_peered":"2024-04-15T15:17:01.073110+0000","last_clean":"2024-04-15T15:17:01.073110+0000","last_became_active":"2024-04-15T15:17:01.066229+0000","last_became_peered":"2024-04-15T15:17:01.066229+0000","last_unstale":"2024-04-15T15:17:01.073110+0000","last_undegraded":"2024-04-15T15:17:01.073110+0000","last_fullsized":"2024-04-15T15:17:01.073110+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.066213+0000","last_change":"2024-04-15T15:17:00.064123+0000","last_active":"2024-04-15T15:17:00.066213+0000","last_peered":"2024-04-15T15:17:00.066213+0000","last_clean":"2024-04-15T15:17:00.066213+0000","last_became_active":"2024-04-15T15:17:00.063535+0000","last_became_peered":"2024-04-15T15:17:00.063535+0000","last_unstale":"2024-04-15T15:17:00.066213+0000","last_undegraded":"2024-04-15T15:17:00.066213+0000","last_fullsized":"2024-04-15T15:17:00.066213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046719+0000","last_change":"2024-04-15T15:17:00.046719+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046719+0000","last_undegraded":"2024-04-15T15:17:00.046719+0000","last_fullsized":"2024-04-15T15:17:00.046719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053705+0000","last_change":"2024-04-15T15:17:00.051113+0000","last_active":"2024-04-15T15:17:00.053705+0000","last_peered":"2024-04-15T15:17:00.053705+0000","last_clean":"2024-04-15T15:17:00.053705+0000","last_became_active":"2024-04-15T15:17:00.050824+0000","last_became_peered":"2024-04-15T15:17:00.050824+0000","last_unstale":"2024-04-15T15:17:00.053705+0000","last_undegraded":"2024-04-15T15:17:00.053705+0000","last_fullsized":"2024-04-15T15:17:00.053705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:50:01.167747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043001+0000","last_change":"2024-04-15T15:17:00.043001+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043001+0000","last_undegraded":"2024-04-15T15:17:00.043001+0000","last_fullsized":"2024-04-15T15:17:00.043001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078424+0000","last_change":"2024-04-15T15:17:00.058333+0000","last_active":"2024-04-15T15:17:01.078424+0000","last_peered":"2024-04-15T15:17:01.078424+0000","last_clean":"2024-04-15T15:17:01.078424+0000","last_became_active":"2024-04-15T15:17:00.057892+0000","last_became_peered":"2024-04-15T15:17:00.057892+0000","last_unstale":"2024-04-15T15:17:01.078424+0000","last_undegraded":"2024-04-15T15:17:01.078424+0000","last_fullsized":"2024-04-15T15:17:01.078424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052245+0000","last_change":"2024-04-15T15:17:00.052245+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052245+0000","last_undegraded":"2024-04-15T15:17:00.052245+0000","last_fullsized":"2024-04-15T15:17:00.052245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053028+0000","last_change":"2024-04-15T15:17:00.049576+0000","last_active":"2024-04-15T15:17:00.053028+0000","last_peered":"2024-04-15T15:17:00.053028+0000","last_clean":"2024-04-15T15:17:00.053028+0000","last_became_active":"2024-04-15T15:17:00.049280+0000","last_became_peered":"2024-04-15T15:17:00.049280+0000","last_unstale":"2024-04-15T15:17:00.053028+0000","last_undegraded":"2024-04-15T15:17:00.053028+0000","last_fullsized":"2024-04-15T15:17:00.053028+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:16:00.136473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046328+0000","last_change":"2024-04-15T15:17:00.046328+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046328+0000","last_undegraded":"2024-04-15T15:17:00.046328+0000","last_fullsized":"2024-04-15T15:17:00.046328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.062211+0000","last_change":"2024-04-15T15:17:00.061061+0000","last_active":"2024-04-15T15:17:00.062211+0000","last_peered":"2024-04-15T15:17:00.062211+0000","last_clean":"2024-04-15T15:17:00.062211+0000","last_became_active":"2024-04-15T15:17:00.060731+0000","last_became_peered":"2024-04-15T15:17:00.060731+0000","last_unstale":"2024-04-15T15:17:00.062211+0000","last_undegraded":"2024-04-15T15:17:00.062211+0000","last_fullsized":"2024-04-15T15:17:00.062211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046258+0000","last_change":"2024-04-15T15:17:00.046258+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046258+0000","last_undegraded":"2024-04-15T15:17:00.046258+0000","last_fullsized":"2024-04-15T15:17:00.046258+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064288+0000","last_change":"2024-04-15T15:17:00.063764+0000","last_active":"2024-04-15T15:17:00.064288+0000","last_peered":"2024-04-15T15:17:00.064288+0000","last_clean":"2024-04-15T15:17:00.064288+0000","last_became_active":"2024-04-15T15:17:00.063528+0000","last_became_peered":"2024-04-15T15:17:00.063528+0000","last_unstale":"2024-04-15T15:17:00.064288+0000","last_undegraded":"2024-04-15T15:17:00.064288+0000","last_fullsized":"2024-04-15T15:17:00.064288+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043848+0000","last_change":"2024-04-15T15:17:00.043848+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043848+0000","last_undegraded":"2024-04-15T15:17:00.043848+0000","last_fullsized":"2024-04-15T15:17:00.043848+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048642+0000","last_change":"2024-04-15T15:17:00.056627+0000","last_active":"2024-04-15T15:17:01.048642+0000","last_peered":"2024-04-15T15:17:01.048642+0000","last_clean":"2024-04-15T15:17:01.048642+0000","last_became_active":"2024-04-15T15:17:00.056317+0000","last_became_peered":"2024-04-15T15:17:00.056317+0000","last_unstale":"2024-04-15T15:17:01.048642+0000","last_undegraded":"2024-04-15T15:17:01.048642+0000","last_fullsized":"2024-04-15T15:17:01.048642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074263+0000","last_change":"2024-04-15T15:17:01.071745+0000","last_active":"2024-04-15T15:17:01.074263+0000","last_peered":"2024-04-15T15:17:01.074263+0000","last_clean":"2024-04-15T15:17:01.074263+0000","last_became_active":"2024-04-15T15:17:01.071466+0000","last_became_peered":"2024-04-15T15:17:01.071466+0000","last_unstale":"2024-04-15T15:17:01.074263+0000","last_undegraded":"2024-04-15T15:17:01.074263+0000","last_fullsized":"2024-04-15T15:17:01.074263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:04.108334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.054345+0000","last_change":"2024-04-15T15:17:00.057863+0000","last_active":"2024-04-15T15:17:01.054345+0000","last_peered":"2024-04-15T15:17:01.054345+0000","last_clean":"2024-04-15T15:17:01.054345+0000","last_became_active":"2024-04-15T15:17:00.057596+0000","last_became_peered":"2024-04-15T15:17:00.057596+0000","last_unstale":"2024-04-15T15:17:01.054345+0000","last_undegraded":"2024-04-15T15:17:01.054345+0000","last_fullsized":"2024-04-15T15:17:01.054345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047469+0000","last_change":"2024-04-15T15:17:00.047469+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047469+0000","last_undegraded":"2024-04-15T15:17:00.047469+0000","last_fullsized":"2024-04-15T15:17:00.047469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.060185+0000","last_change":"2024-04-15T15:17:00.058811+0000","last_active":"2024-04-15T15:17:00.060185+0000","last_peered":"2024-04-15T15:17:00.060185+0000","last_clean":"2024-04-15T15:17:00.060185+0000","last_became_active":"2024-04-15T15:17:00.058530+0000","last_became_peered":"2024-04-15T15:17:00.058530+0000","last_unstale":"2024-04-15T15:17:00.060185+0000","last_undegraded":"2024-04-15T15:17:00.060185+0000","last_fullsized":"2024-04-15T15:17:00.060185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:03.103510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074626+0000","last_change":"2024-04-15T15:17:01.070460+0000","last_active":"2024-04-15T15:17:01.074626+0000","last_peered":"2024-04-15T15:17:01.074626+0000","last_clean":"2024-04-15T15:17:01.074626+0000","last_became_active":"2024-04-15T15:17:01.070110+0000","last_became_peered":"2024-04-15T15:17:01.070110+0000","last_unstale":"2024-04-15T15:17:01.074626+0000","last_undegraded":"2024-04-15T15:17:01.074626+0000","last_fullsized":"2024-04-15T15:17:01.074626+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064075+0000","last_change":"2024-04-15T15:17:00.061284+0000","last_active":"2024-04-15T15:17:00.064075+0000","last_peered":"2024-04-15T15:17:00.064075+0000","last_clean":"2024-04-15T15:17:00.064075+0000","last_became_active":"2024-04-15T15:17:00.060520+0000","last_became_peered":"2024-04-15T15:17:00.060520+0000","last_unstale":"2024-04-15T15:17:00.064075+0000","last_undegraded":"2024-04-15T15:17:00.064075+0000","last_fullsized":"2024-04-15T15:17:00.064075+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053524+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:00.053524+0000","last_peered":"2024-04-15T15:17:00.053524+0000","last_clean":"2024-04-15T15:17:00.053524+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:00.053524+0000","last_undegraded":"2024-04-15T15:17:00.053524+0000","last_fullsized":"2024-04-15T15:17:00.053524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":187741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1015808,"data_stored":770360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":179992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":173325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":179992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":763693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:05.456 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:05.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.593+0000 7fbe0f91c640 1 Processor -- start 2024-04-15T15:17:05.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.594+0000 7fbe0f91c640 1 -- start start 2024-04-15T15:17:05.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.594+0000 7fbe0f91c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbe080d0690 0x7fbe080d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.594+0000 7fbe0f91c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe080d0fb0 con 0x7fbe080b3160 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.594+0000 7fbe0f91c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0815cb50 con 0x7fbe080ae280 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.594+0000 7fbe0f91c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0815bda0 con 0x7fbe080d0690 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe0ceb7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbe080b3160 0x7fbe0815b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52246/0 (socket says 172.21.15.46:52246) 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe0ceb7640 1 -- 172.21.15.46:0/717254129 learned_addr learned my addr 172.21.15.46:0/717254129 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe0d6b8640 1 --2- 172.21.15.46:0/717254129 >> v2:172.21.15.78:3300/0 conn(0x7fbe080d0690 0x7fbe080d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790187686 0 0) 0x7fbe080d0fb0 con 0x7fbe080b3160 2024-04-15T15:17:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbdf0003520 con 0x7fbe080b3160 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195194673 0 0) 0x7fbe0815cb50 con 0x7fbe080ae280 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe080d0fb0 con 0x7fbe080ae280 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe0d6b8640 1 -- 172.21.15.46:0/717254129 >> v1:172.21.15.78:6789/0 conn(0x7fbe080ae280 legacy=0x7fbe080ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe0d6b8640 1 -- 172.21.15.46:0/717254129 >> v1:172.21.15.46:6789/0 conn(0x7fbe080b3160 legacy=0x7fbe0815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2254028740 0 0) 0x7fbdf0003520 con 0x7fbe080b3160 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.595+0000 7fbe0d6b8640 1 -- 172.21.15.46:0/717254129 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0815bfd0 con 0x7fbe080d0690 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.596+0000 7fbe0d6b8640 1 --2- 172.21.15.46:0/717254129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080d0690 0x7fbe080d0a70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe000097a0 tx=0x7fbe0002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e36e573134a3799 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.596+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00038070 con 0x7fbe080d0690 2024-04-15T15:17:05.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.596+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe00031500 con 0x7fbe080d0690 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.596+0000 7fbe07fff640 1 -- 172.21.15.46:0/717254129 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc001670 con 0x7fbe080d0690 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 -- 172.21.15.46:0/717254129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080d0690 msgr2=0x7fbe080d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/717254129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080d0690 0x7fbe080d0a70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbe000097a0 tx=0x7fbe0002b820 comp rx=0 tx=0).stop 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 -- 172.21.15.46:0/717254129 shutdown_connections 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/717254129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080d0690 0x7fbe080d0a70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 -- 172.21.15.46:0/717254129 >> 172.21.15.46:0/717254129 conn(0x7fbe0806be00 msgr2=0x7fbe0806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 -- 172.21.15.46:0/717254129 shutdown_connections 2024-04-15T15:17:05.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.597+0000 7fbe0f91c640 1 -- 172.21.15.46:0/717254129 wait complete. 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.598+0000 7fbe0f91c640 1 Processor -- start 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.598+0000 7fbe0f91c640 1 -- start start 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0f91c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe080ae280 0x7fbe0809db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0f91c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080b3160 0x7fbe0815ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0f91c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 0x7fbe081569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0f91c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0815c870 con 0x7fbe080d0690 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0f91c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0815c590 con 0x7fbe080b3160 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0f91c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbe080b4fb0 con 0x7fbe080ae280 2024-04-15T15:17:05.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 0x7fbe081569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 0x7fbe081569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48176/0 (socket says 172.21.15.46:48176) 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 -- 172.21.15.46:0/3565442227 learned_addr learned my addr 172.21.15.46:0/3565442227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0d6b8640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe080ae280 0x7fbe0809db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0ceb7640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080b3160 0x7fbe0815ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 -- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe080ae280 msgr2=0x7fbe0809db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe080ae280 0x7fbe0809db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 -- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080b3160 msgr2=0x7fbe0815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080b3160 0x7fbe0815ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 -- 172.21.15.46:0/3565442227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0809a340 con 0x7fbe080d0690 2024-04-15T15:17:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0d6b8640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe080ae280 0x7fbe0809db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe0deb9640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 0x7fbe081569a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800c6e0 tx=0x7fbdf800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.599+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8003b70 con 0x7fbe080d0690 2024-04-15T15:17:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.600+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0809a630 con 0x7fbe080d0690 2024-04-15T15:17:05.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.600+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf8003d10 con 0x7fbe080d0690 2024-04-15T15:17:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.600+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf80052d0 con 0x7fbe080d0690 2024-04-15T15:17:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.600+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0809ae70 con 0x7fbe080d0690 2024-04-15T15:17:05.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.601+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe0809b070 con 0x7fbe080d0690 2024-04-15T15:17:05.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.601+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbdf8005470 con 0x7fbe080d0690 2024-04-15T15:17:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.602+0000 7fbe05ffb640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf406c8f0 0x7fbdf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.602+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbdf808bc50 con 0x7fbe080d0690 2024-04-15T15:17:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.602+0000 7fbe0d6b8640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf406c8f0 0x7fbdf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.603+0000 7fbe0d6b8640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf406c8f0 0x7fbdf406edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbe00035000 tx=0x7fbe00008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.606+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe0809b070 con 0x7fbe080d0690 2024-04-15T15:17:05.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.790+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbe080689e0 con 0x7fbdf406c8f0 2024-04-15T15:17:05.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.796+0000 7fbe05ffb640 1 -- 172.21.15.46:0/3565442227 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248614 (secure 0 0 0) 0x7fbe080689e0 con 0x7fbdf406c8f0 2024-04-15T15:17:05.801 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:05.802 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf406c8f0 msgr2=0x7fbdf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf406c8f0 0x7fbdf406edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbe00035000 tx=0x7fbe00008030 comp rx=0 tx=0).stop 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 msgr2=0x7fbe081569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 0x7fbe081569a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800c6e0 tx=0x7fbdf800cba0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 shutdown_connections 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbdf406c8f0 0x7fbdf406edb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe080d0690 0x7fbe081569a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe080b3160 0x7fbe0815ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 --2- 172.21.15.46:0/3565442227 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbe080ae280 0x7fbe0809db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 >> 172.21.15.46:0/3565442227 conn(0x7fbe0806be00 msgr2=0x7fbe08071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 shutdown_connections 2024-04-15T15:17:05.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.800+0000 7fbe0f91c640 1 -- 172.21.15.46:0/3565442227 wait complete. 2024-04-15T15:17:05.812 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-15T15:17:04.373266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237304,"kb_used_data":4488,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737416,"statfs":{"total":773094113280,"available":772851113984,"internally_reserved":0,"allocated":4595712,"data_stored":2630137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.338094"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048962+0000","last_change":"2024-04-15T15:17:00.048962+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048962+0000","last_undegraded":"2024-04-15T15:17:00.048962+0000","last_fullsized":"2024-04-15T15:17:00.048962+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046889+0000","last_change":"2024-04-15T15:17:00.046889+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046889+0000","last_undegraded":"2024-04-15T15:17:00.046889+0000","last_fullsized":"2024-04-15T15:17:00.046889+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047885+0000","last_change":"2024-04-15T15:17:00.047885+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047885+0000","last_undegraded":"2024-04-15T15:17:00.047885+0000","last_fullsized":"2024-04-15T15:17:00.047885+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045994+0000","last_change":"2024-04-15T15:17:00.045994+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045994+0000","last_undegraded":"2024-04-15T15:17:00.045994+0000","last_fullsized":"2024-04-15T15:17:00.045994+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043764+0000","last_change":"2024-04-15T15:17:00.043764+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043764+0000","last_undegraded":"2024-04-15T15:17:00.043764+0000","last_fullsized":"2024-04-15T15:17:00.043764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044975+0000","last_change":"2024-04-15T15:17:00.044975+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044975+0000","last_undegraded":"2024-04-15T15:17:00.044975+0000","last_fullsized":"2024-04-15T15:17:00.044975+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074681+0000","last_change":"2024-04-15T15:17:01.069002+0000","last_active":"2024-04-15T15:17:01.074681+0000","last_peered":"2024-04-15T15:17:01.074681+0000","last_clean":"2024-04-15T15:17:01.074681+0000","last_became_active":"2024-04-15T15:17:01.068755+0000","last_became_peered":"2024-04-15T15:17:01.068755+0000","last_unstale":"2024-04-15T15:17:01.074681+0000","last_undegraded":"2024-04-15T15:17:01.074681+0000","last_fullsized":"2024-04-15T15:17:01.074681+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.049417+0000","last_change":"2024-04-15T15:17:00.049417+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.049417+0000","last_undegraded":"2024-04-15T15:17:00.049417+0000","last_fullsized":"2024-04-15T15:17:00.049417+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.049028+0000","last_change":"2024-04-15T15:17:00.049028+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.049028+0000","last_undegraded":"2024-04-15T15:17:00.049028+0000","last_fullsized":"2024-04-15T15:17:00.049028+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073089+0000","last_change":"2024-04-15T15:17:01.068258+0000","last_active":"2024-04-15T15:17:01.073089+0000","last_peered":"2024-04-15T15:17:01.073089+0000","last_clean":"2024-04-15T15:17:01.073089+0000","last_became_active":"2024-04-15T15:17:01.067888+0000","last_became_peered":"2024-04-15T15:17:01.067888+0000","last_unstale":"2024-04-15T15:17:01.073089+0000","last_undegraded":"2024-04-15T15:17:01.073089+0000","last_fullsized":"2024-04-15T15:17:01.073089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075397+0000","last_change":"2024-04-15T15:17:01.070976+0000","last_active":"2024-04-15T15:17:01.075397+0000","last_peered":"2024-04-15T15:17:01.075397+0000","last_clean":"2024-04-15T15:17:01.075397+0000","last_became_active":"2024-04-15T15:17:01.070678+0000","last_became_peered":"2024-04-15T15:17:01.070678+0000","last_unstale":"2024-04-15T15:17:01.075397+0000","last_undegraded":"2024-04-15T15:17:01.075397+0000","last_fullsized":"2024-04-15T15:17:01.075397+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:38:18.560062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045186+0000","last_change":"2024-04-15T15:17:00.045186+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045186+0000","last_undegraded":"2024-04-15T15:17:00.045186+0000","last_fullsized":"2024-04-15T15:17:00.045186+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.065827+0000","last_change":"2024-04-15T15:17:01.064962+0000","last_active":"2024-04-15T15:17:01.065827+0000","last_peered":"2024-04-15T15:17:01.065827+0000","last_clean":"2024-04-15T15:17:01.065827+0000","last_became_active":"2024-04-15T15:17:01.064395+0000","last_became_peered":"2024-04-15T15:17:01.064395+0000","last_unstale":"2024-04-15T15:17:01.065827+0000","last_undegraded":"2024-04-15T15:17:01.065827+0000","last_fullsized":"2024-04-15T15:17:01.065827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072393+0000","last_change":"2024-04-15T15:17:01.069059+0000","last_active":"2024-04-15T15:17:01.072393+0000","last_peered":"2024-04-15T15:17:01.072393+0000","last_clean":"2024-04-15T15:17:01.072393+0000","last_became_active":"2024-04-15T15:17:01.067835+0000","last_became_peered":"2024-04-15T15:17:01.067835+0000","last_unstale":"2024-04-15T15:17:01.072393+0000","last_undegraded":"2024-04-15T15:17:01.072393+0000","last_fullsized":"2024-04-15T15:17:01.072393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073052+0000","last_change":"2024-04-15T15:17:01.065939+0000","last_active":"2024-04-15T15:17:01.073052+0000","last_peered":"2024-04-15T15:17:01.073052+0000","last_clean":"2024-04-15T15:17:01.073052+0000","last_became_active":"2024-04-15T15:17:01.065641+0000","last_became_peered":"2024-04-15T15:17:01.065641+0000","last_unstale":"2024-04-15T15:17:01.073052+0000","last_undegraded":"2024-04-15T15:17:01.073052+0000","last_fullsized":"2024-04-15T15:17:01.073052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052205+0000","last_change":"2024-04-15T15:17:00.052205+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052205+0000","last_undegraded":"2024-04-15T15:17:00.052205+0000","last_fullsized":"2024-04-15T15:17:00.052205+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047966+0000","last_change":"2024-04-15T15:17:00.047966+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047966+0000","last_undegraded":"2024-04-15T15:17:00.047966+0000","last_fullsized":"2024-04-15T15:17:00.047966+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073211+0000","last_change":"2024-04-15T15:17:01.070470+0000","last_active":"2024-04-15T15:17:01.073211+0000","last_peered":"2024-04-15T15:17:01.073211+0000","last_clean":"2024-04-15T15:17:01.073211+0000","last_became_active":"2024-04-15T15:17:01.070200+0000","last_became_peered":"2024-04-15T15:17:01.070200+0000","last_unstale":"2024-04-15T15:17:01.073211+0000","last_undegraded":"2024-04-15T15:17:01.073211+0000","last_fullsized":"2024-04-15T15:17:01.073211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043158+0000","last_change":"2024-04-15T15:17:00.043158+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043158+0000","last_undegraded":"2024-04-15T15:17:00.043158+0000","last_fullsized":"2024-04-15T15:17:00.043158+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.076558+0000","last_change":"2024-04-15T15:17:01.071687+0000","last_active":"2024-04-15T15:17:01.076558+0000","last_peered":"2024-04-15T15:17:01.076558+0000","last_clean":"2024-04-15T15:17:01.076558+0000","last_became_active":"2024-04-15T15:17:01.071330+0000","last_became_peered":"2024-04-15T15:17:01.071330+0000","last_unstale":"2024-04-15T15:17:01.076558+0000","last_undegraded":"2024-04-15T15:17:01.076558+0000","last_fullsized":"2024-04-15T15:17:01.076558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073107+0000","last_change":"2024-04-15T15:17:01.067759+0000","last_active":"2024-04-15T15:17:01.073107+0000","last_peered":"2024-04-15T15:17:01.073107+0000","last_clean":"2024-04-15T15:17:01.073107+0000","last_became_active":"2024-04-15T15:17:01.067476+0000","last_became_peered":"2024-04-15T15:17:01.067476+0000","last_unstale":"2024-04-15T15:17:01.073107+0000","last_undegraded":"2024-04-15T15:17:01.073107+0000","last_fullsized":"2024-04-15T15:17:01.073107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072415+0000","last_change":"2024-04-15T15:17:01.070298+0000","last_active":"2024-04-15T15:17:01.072415+0000","last_peered":"2024-04-15T15:17:01.072415+0000","last_clean":"2024-04-15T15:17:01.072415+0000","last_became_active":"2024-04-15T15:17:01.070032+0000","last_became_peered":"2024-04-15T15:17:01.070032+0000","last_unstale":"2024-04-15T15:17:01.072415+0000","last_undegraded":"2024-04-15T15:17:01.072415+0000","last_fullsized":"2024-04-15T15:17:01.072415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:23.034253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052838+0000","last_change":"2024-04-15T15:17:00.052838+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052838+0000","last_undegraded":"2024-04-15T15:17:00.052838+0000","last_fullsized":"2024-04-15T15:17:00.052838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.042458+0000","last_change":"2024-04-15T15:17:00.042458+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.042458+0000","last_undegraded":"2024-04-15T15:17:00.042458+0000","last_fullsized":"2024-04-15T15:17:00.042458+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061210+0000","last_change":"2024-04-15T15:17:01.056648+0000","last_active":"2024-04-15T15:17:01.061210+0000","last_peered":"2024-04-15T15:17:01.061210+0000","last_clean":"2024-04-15T15:17:01.061210+0000","last_became_active":"2024-04-15T15:17:01.056354+0000","last_became_peered":"2024-04-15T15:17:01.056354+0000","last_unstale":"2024-04-15T15:17:01.061210+0000","last_undegraded":"2024-04-15T15:17:01.061210+0000","last_fullsized":"2024-04-15T15:17:01.061210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044860+0000","last_change":"2024-04-15T15:17:00.044860+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044860+0000","last_undegraded":"2024-04-15T15:17:00.044860+0000","last_fullsized":"2024-04-15T15:17:00.044860+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.062082+0000","last_change":"2024-04-15T15:17:01.059015+0000","last_active":"2024-04-15T15:17:01.062082+0000","last_peered":"2024-04-15T15:17:01.062082+0000","last_clean":"2024-04-15T15:17:01.062082+0000","last_became_active":"2024-04-15T15:17:01.058718+0000","last_became_peered":"2024-04-15T15:17:01.058718+0000","last_unstale":"2024-04-15T15:17:01.062082+0000","last_undegraded":"2024-04-15T15:17:01.062082+0000","last_fullsized":"2024-04-15T15:17:01.062082+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074673+0000","last_change":"2024-04-15T15:17:01.069694+0000","last_active":"2024-04-15T15:17:01.074673+0000","last_peered":"2024-04-15T15:17:01.074673+0000","last_clean":"2024-04-15T15:17:01.074673+0000","last_became_active":"2024-04-15T15:17:01.069408+0000","last_became_peered":"2024-04-15T15:17:01.069408+0000","last_unstale":"2024-04-15T15:17:01.074673+0000","last_undegraded":"2024-04-15T15:17:01.074673+0000","last_fullsized":"2024-04-15T15:17:01.074673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072620+0000","last_change":"2024-04-15T15:17:01.068919+0000","last_active":"2024-04-15T15:17:01.072620+0000","last_peered":"2024-04-15T15:17:01.072620+0000","last_clean":"2024-04-15T15:17:01.072620+0000","last_became_active":"2024-04-15T15:17:01.068639+0000","last_became_peered":"2024-04-15T15:17:01.068639+0000","last_unstale":"2024-04-15T15:17:01.072620+0000","last_undegraded":"2024-04-15T15:17:01.072620+0000","last_fullsized":"2024-04-15T15:17:01.072620+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061418+0000","last_change":"2024-04-15T15:17:01.056050+0000","last_active":"2024-04-15T15:17:01.061418+0000","last_peered":"2024-04-15T15:17:01.061418+0000","last_clean":"2024-04-15T15:17:01.061418+0000","last_became_active":"2024-04-15T15:17:01.055735+0000","last_became_peered":"2024-04-15T15:17:01.055735+0000","last_unstale":"2024-04-15T15:17:01.061418+0000","last_undegraded":"2024-04-15T15:17:01.061418+0000","last_fullsized":"2024-04-15T15:17:01.061418+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:51.299505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048125+0000","last_change":"2024-04-15T15:17:00.048125+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048125+0000","last_undegraded":"2024-04-15T15:17:00.048125+0000","last_fullsized":"2024-04-15T15:17:00.048125+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.044106+0000","last_change":"2024-04-15T15:17:00.044106+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.044106+0000","last_undegraded":"2024-04-15T15:17:00.044106+0000","last_fullsized":"2024-04-15T15:17:00.044106+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047283+0000","last_change":"2024-04-15T15:17:00.047283+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047283+0000","last_undegraded":"2024-04-15T15:17:00.047283+0000","last_fullsized":"2024-04-15T15:17:00.047283+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078586+0000","last_change":"2024-04-15T15:17:00.053485+0000","last_active":"2024-04-15T15:17:01.078586+0000","last_peered":"2024-04-15T15:17:01.078586+0000","last_clean":"2024-04-15T15:17:01.078586+0000","last_became_active":"2024-04-15T15:17:00.053211+0000","last_became_peered":"2024-04-15T15:17:00.053211+0000","last_unstale":"2024-04-15T15:17:01.078586+0000","last_undegraded":"2024-04-15T15:17:01.078586+0000","last_fullsized":"2024-04-15T15:17:01.078586+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043633+0000","last_change":"2024-04-15T15:17:00.043633+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043633+0000","last_undegraded":"2024-04-15T15:17:00.043633+0000","last_fullsized":"2024-04-15T15:17:00.043633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058716+0000","last_change":"2024-04-15T15:17:00.056209+0000","last_active":"2024-04-15T15:17:00.058716+0000","last_peered":"2024-04-15T15:17:00.058716+0000","last_clean":"2024-04-15T15:17:00.058716+0000","last_became_active":"2024-04-15T15:17:00.055881+0000","last_became_peered":"2024-04-15T15:17:00.055881+0000","last_unstale":"2024-04-15T15:17:00.058716+0000","last_undegraded":"2024-04-15T15:17:00.058716+0000","last_fullsized":"2024-04-15T15:17:00.058716+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072065+0000","last_change":"2024-04-15T15:17:01.067226+0000","last_active":"2024-04-15T15:17:01.072065+0000","last_peered":"2024-04-15T15:17:01.072065+0000","last_clean":"2024-04-15T15:17:01.072065+0000","last_became_active":"2024-04-15T15:17:01.066479+0000","last_became_peered":"2024-04-15T15:17:01.066479+0000","last_unstale":"2024-04-15T15:17:01.072065+0000","last_undegraded":"2024-04-15T15:17:01.072065+0000","last_fullsized":"2024-04-15T15:17:01.072065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058483+0000","last_change":"2024-04-15T15:17:00.054251+0000","last_active":"2024-04-15T15:17:00.058483+0000","last_peered":"2024-04-15T15:17:00.058483+0000","last_clean":"2024-04-15T15:17:00.058483+0000","last_became_active":"2024-04-15T15:17:00.053980+0000","last_became_peered":"2024-04-15T15:17:00.053980+0000","last_unstale":"2024-04-15T15:17:00.058483+0000","last_undegraded":"2024-04-15T15:17:00.058483+0000","last_fullsized":"2024-04-15T15:17:00.058483+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.050290+0000","last_change":"2024-04-15T15:17:00.050290+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.050290+0000","last_undegraded":"2024-04-15T15:17:00.050290+0000","last_fullsized":"2024-04-15T15:17:00.050290+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.080694+0000","last_change":"2024-04-15T15:17:00.058378+0000","last_active":"2024-04-15T15:17:01.080694+0000","last_peered":"2024-04-15T15:17:01.080694+0000","last_clean":"2024-04-15T15:17:01.080694+0000","last_became_active":"2024-04-15T15:17:00.058024+0000","last_became_peered":"2024-04-15T15:17:00.058024+0000","last_unstale":"2024-04-15T15:17:01.080694+0000","last_undegraded":"2024-04-15T15:17:01.080694+0000","last_fullsized":"2024-04-15T15:17:01.080694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047108+0000","last_change":"2024-04-15T15:17:00.047108+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047108+0000","last_undegraded":"2024-04-15T15:17:00.047108+0000","last_fullsized":"2024-04-15T15:17:00.047108+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.077676+0000","last_change":"2024-04-15T15:17:00.052588+0000","last_active":"2024-04-15T15:17:01.077676+0000","last_peered":"2024-04-15T15:17:01.077676+0000","last_clean":"2024-04-15T15:17:01.077676+0000","last_became_active":"2024-04-15T15:17:00.052287+0000","last_became_peered":"2024-04-15T15:17:00.052287+0000","last_unstale":"2024-04-15T15:17:01.077676+0000","last_undegraded":"2024-04-15T15:17:01.077676+0000","last_fullsized":"2024-04-15T15:17:01.077676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079604+0000","last_change":"2024-04-15T15:17:01.077529+0000","last_active":"2024-04-15T15:17:01.079604+0000","last_peered":"2024-04-15T15:17:01.079604+0000","last_clean":"2024-04-15T15:17:01.079604+0000","last_became_active":"2024-04-15T15:17:01.077231+0000","last_became_peered":"2024-04-15T15:17:01.077231+0000","last_unstale":"2024-04-15T15:17:01.079604+0000","last_undegraded":"2024-04-15T15:17:01.079604+0000","last_fullsized":"2024-04-15T15:17:01.079604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:58:51.072457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.063827+0000","last_change":"2024-04-15T15:17:00.060924+0000","last_active":"2024-04-15T15:17:00.063827+0000","last_peered":"2024-04-15T15:17:00.063827+0000","last_clean":"2024-04-15T15:17:00.063827+0000","last_became_active":"2024-04-15T15:17:00.060610+0000","last_became_peered":"2024-04-15T15:17:00.060610+0000","last_unstale":"2024-04-15T15:17:00.063827+0000","last_undegraded":"2024-04-15T15:17:00.063827+0000","last_fullsized":"2024-04-15T15:17:00.063827+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072370+0000","last_change":"2024-04-15T15:17:01.068809+0000","last_active":"2024-04-15T15:17:01.072370+0000","last_peered":"2024-04-15T15:17:01.072370+0000","last_clean":"2024-04-15T15:17:01.072370+0000","last_became_active":"2024-04-15T15:17:01.067740+0000","last_became_peered":"2024-04-15T15:17:01.067740+0000","last_unstale":"2024-04-15T15:17:01.072370+0000","last_undegraded":"2024-04-15T15:17:01.072370+0000","last_fullsized":"2024-04-15T15:17:01.072370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058130+0000","last_change":"2024-04-15T15:17:00.056079+0000","last_active":"2024-04-15T15:17:00.058130+0000","last_peered":"2024-04-15T15:17:00.058130+0000","last_clean":"2024-04-15T15:17:00.058130+0000","last_became_active":"2024-04-15T15:17:00.055794+0000","last_became_peered":"2024-04-15T15:17:00.055794+0000","last_unstale":"2024-04-15T15:17:00.058130+0000","last_undegraded":"2024-04-15T15:17:00.058130+0000","last_fullsized":"2024-04-15T15:17:00.058130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.050788+0000","last_change":"2024-04-15T15:17:00.050788+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.050788+0000","last_undegraded":"2024-04-15T15:17:00.050788+0000","last_fullsized":"2024-04-15T15:17:00.050788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056679+0000","last_change":"2024-04-15T15:17:00.048501+0000","last_active":"2024-04-15T15:17:01.056679+0000","last_peered":"2024-04-15T15:17:01.056679+0000","last_clean":"2024-04-15T15:17:01.056679+0000","last_became_active":"2024-04-15T15:17:00.048148+0000","last_became_peered":"2024-04-15T15:17:00.048148+0000","last_unstale":"2024-04-15T15:17:01.056679+0000","last_undegraded":"2024-04-15T15:17:01.056679+0000","last_fullsized":"2024-04-15T15:17:01.056679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043571+0000","last_change":"2024-04-15T15:17:00.043571+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043571+0000","last_undegraded":"2024-04-15T15:17:00.043571+0000","last_fullsized":"2024-04-15T15:17:00.043571+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.060211+0000","last_change":"2024-04-15T15:17:00.059059+0000","last_active":"2024-04-15T15:17:00.060211+0000","last_peered":"2024-04-15T15:17:00.060211+0000","last_clean":"2024-04-15T15:17:00.060211+0000","last_became_active":"2024-04-15T15:17:00.058527+0000","last_became_peered":"2024-04-15T15:17:00.058527+0000","last_unstale":"2024-04-15T15:17:00.060211+0000","last_undegraded":"2024-04-15T15:17:00.060211+0000","last_fullsized":"2024-04-15T15:17:00.060211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052662+0000","last_change":"2024-04-15T15:17:00.052662+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052662+0000","last_undegraded":"2024-04-15T15:17:00.052662+0000","last_fullsized":"2024-04-15T15:17:00.052662+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056094+0000","last_change":"2024-04-15T15:17:00.058594+0000","last_active":"2024-04-15T15:17:01.056094+0000","last_peered":"2024-04-15T15:17:01.056094+0000","last_clean":"2024-04-15T15:17:01.056094+0000","last_became_active":"2024-04-15T15:17:00.058359+0000","last_became_peered":"2024-04-15T15:17:00.058359+0000","last_unstale":"2024-04-15T15:17:01.056094+0000","last_undegraded":"2024-04-15T15:17:01.056094+0000","last_fullsized":"2024-04-15T15:17:01.056094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.045457+0000","last_change":"2024-04-15T15:17:00.045457+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.045457+0000","last_undegraded":"2024-04-15T15:17:00.045457+0000","last_fullsized":"2024-04-15T15:17:00.045457+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078903+0000","last_change":"2024-04-15T15:17:00.055108+0000","last_active":"2024-04-15T15:17:01.078903+0000","last_peered":"2024-04-15T15:17:01.078903+0000","last_clean":"2024-04-15T15:17:01.078903+0000","last_became_active":"2024-04-15T15:17:00.054818+0000","last_became_peered":"2024-04-15T15:17:00.054818+0000","last_unstale":"2024-04-15T15:17:01.078903+0000","last_undegraded":"2024-04-15T15:17:01.078903+0000","last_fullsized":"2024-04-15T15:17:01.078903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072041+0000","last_change":"2024-04-15T15:17:01.066511+0000","last_active":"2024-04-15T15:17:01.072041+0000","last_peered":"2024-04-15T15:17:01.072041+0000","last_clean":"2024-04-15T15:17:01.072041+0000","last_became_active":"2024-04-15T15:17:01.066201+0000","last_became_peered":"2024-04-15T15:17:01.066201+0000","last_unstale":"2024-04-15T15:17:01.072041+0000","last_undegraded":"2024-04-15T15:17:01.072041+0000","last_fullsized":"2024-04-15T15:17:01.072041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.062191+0000","last_change":"2024-04-15T15:17:00.061109+0000","last_active":"2024-04-15T15:17:00.062191+0000","last_peered":"2024-04-15T15:17:00.062191+0000","last_clean":"2024-04-15T15:17:00.062191+0000","last_became_active":"2024-04-15T15:17:00.060823+0000","last_became_peered":"2024-04-15T15:17:00.060823+0000","last_unstale":"2024-04-15T15:17:00.062191+0000","last_undegraded":"2024-04-15T15:17:00.062191+0000","last_fullsized":"2024-04-15T15:17:00.062191+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046672+0000","last_change":"2024-04-15T15:17:00.046672+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046672+0000","last_undegraded":"2024-04-15T15:17:00.046672+0000","last_fullsized":"2024-04-15T15:17:00.046672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058974+0000","last_change":"2024-04-15T15:17:00.057661+0000","last_active":"2024-04-15T15:17:00.058974+0000","last_peered":"2024-04-15T15:17:00.058974+0000","last_clean":"2024-04-15T15:17:00.058974+0000","last_became_active":"2024-04-15T15:17:00.057376+0000","last_became_peered":"2024-04-15T15:17:00.057376+0000","last_unstale":"2024-04-15T15:17:00.058974+0000","last_undegraded":"2024-04-15T15:17:00.058974+0000","last_fullsized":"2024-04-15T15:17:00.058974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:22.029429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048446+0000","last_change":"2024-04-15T15:17:00.048446+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048446+0000","last_undegraded":"2024-04-15T15:17:00.048446+0000","last_fullsized":"2024-04-15T15:17:00.048446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.058358+0000","last_change":"2024-04-15T15:17:00.055785+0000","last_active":"2024-04-15T15:17:00.058358+0000","last_peered":"2024-04-15T15:17:00.058358+0000","last_clean":"2024-04-15T15:17:00.058358+0000","last_became_active":"2024-04-15T15:17:00.055509+0000","last_became_peered":"2024-04-15T15:17:00.055509+0000","last_unstale":"2024-04-15T15:17:00.058358+0000","last_undegraded":"2024-04-15T15:17:00.058358+0000","last_fullsized":"2024-04-15T15:17:00.058358+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047149+0000","last_change":"2024-04-15T15:17:00.047149+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047149+0000","last_undegraded":"2024-04-15T15:17:00.047149+0000","last_fullsized":"2024-04-15T15:17:00.047149+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048522+0000","last_change":"2024-04-15T15:17:00.055842+0000","last_active":"2024-04-15T15:17:01.048522+0000","last_peered":"2024-04-15T15:17:01.048522+0000","last_clean":"2024-04-15T15:17:01.048522+0000","last_became_active":"2024-04-15T15:17:00.054053+0000","last_became_peered":"2024-04-15T15:17:00.054053+0000","last_unstale":"2024-04-15T15:17:01.048522+0000","last_undegraded":"2024-04-15T15:17:01.048522+0000","last_fullsized":"2024-04-15T15:17:01.048522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047661+0000","last_change":"2024-04-15T15:17:00.047661+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047661+0000","last_undegraded":"2024-04-15T15:17:00.047661+0000","last_fullsized":"2024-04-15T15:17:00.047661+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064003+0000","last_change":"2024-04-15T15:17:00.061010+0000","last_active":"2024-04-15T15:17:00.064003+0000","last_peered":"2024-04-15T15:17:00.064003+0000","last_clean":"2024-04-15T15:17:00.064003+0000","last_became_active":"2024-04-15T15:17:00.060723+0000","last_became_peered":"2024-04-15T15:17:00.060723+0000","last_unstale":"2024-04-15T15:17:00.064003+0000","last_undegraded":"2024-04-15T15:17:00.064003+0000","last_fullsized":"2024-04-15T15:17:00.064003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.051279+0000","last_change":"2024-04-15T15:17:00.051279+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.051279+0000","last_undegraded":"2024-04-15T15:17:00.051279+0000","last_fullsized":"2024-04-15T15:17:00.051279+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078087+0000","last_change":"2024-04-15T15:17:00.053088+0000","last_active":"2024-04-15T15:17:01.078087+0000","last_peered":"2024-04-15T15:17:01.078087+0000","last_clean":"2024-04-15T15:17:01.078087+0000","last_became_active":"2024-04-15T15:17:00.052762+0000","last_became_peered":"2024-04-15T15:17:00.052762+0000","last_unstale":"2024-04-15T15:17:01.078087+0000","last_undegraded":"2024-04-15T15:17:01.078087+0000","last_fullsized":"2024-04-15T15:17:01.078087+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075375+0000","last_change":"2024-04-15T15:17:01.067435+0000","last_active":"2024-04-15T15:17:01.075375+0000","last_peered":"2024-04-15T15:17:01.075375+0000","last_clean":"2024-04-15T15:17:01.075375+0000","last_became_active":"2024-04-15T15:17:01.066970+0000","last_became_peered":"2024-04-15T15:17:01.066970+0000","last_unstale":"2024-04-15T15:17:01.075375+0000","last_undegraded":"2024-04-15T15:17:01.075375+0000","last_fullsized":"2024-04-15T15:17:01.075375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079013+0000","last_change":"2024-04-15T15:17:00.049730+0000","last_active":"2024-04-15T15:17:01.079013+0000","last_peered":"2024-04-15T15:17:01.079013+0000","last_clean":"2024-04-15T15:17:01.079013+0000","last_became_active":"2024-04-15T15:17:00.049472+0000","last_became_peered":"2024-04-15T15:17:00.049472+0000","last_unstale":"2024-04-15T15:17:01.079013+0000","last_undegraded":"2024-04-15T15:17:01.079013+0000","last_fullsized":"2024-04-15T15:17:01.079013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072086+0000","last_change":"2024-04-15T15:17:01.067474+0000","last_active":"2024-04-15T15:17:01.072086+0000","last_peered":"2024-04-15T15:17:01.072086+0000","last_clean":"2024-04-15T15:17:01.072086+0000","last_became_active":"2024-04-15T15:17:01.066802+0000","last_became_peered":"2024-04-15T15:17:01.066802+0000","last_unstale":"2024-04-15T15:17:01.072086+0000","last_undegraded":"2024-04-15T15:17:01.072086+0000","last_fullsized":"2024-04-15T15:17:01.072086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.055389+0000","last_change":"2024-04-15T15:17:00.054252+0000","last_active":"2024-04-15T15:17:00.055389+0000","last_peered":"2024-04-15T15:17:00.055389+0000","last_clean":"2024-04-15T15:17:00.055389+0000","last_became_active":"2024-04-15T15:17:00.053917+0000","last_became_peered":"2024-04-15T15:17:00.053917+0000","last_unstale":"2024-04-15T15:17:00.055389+0000","last_undegraded":"2024-04-15T15:17:00.055389+0000","last_fullsized":"2024-04-15T15:17:00.055389+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:53:53.053021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043001+0000","last_change":"2024-04-15T15:17:00.043001+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043001+0000","last_undegraded":"2024-04-15T15:17:00.043001+0000","last_fullsized":"2024-04-15T15:17:00.043001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.057998+0000","last_change":"2024-04-15T15:17:00.053573+0000","last_active":"2024-04-15T15:17:00.057998+0000","last_peered":"2024-04-15T15:17:00.057998+0000","last_clean":"2024-04-15T15:17:00.057998+0000","last_became_active":"2024-04-15T15:17:00.053296+0000","last_became_peered":"2024-04-15T15:17:00.053296+0000","last_unstale":"2024-04-15T15:17:00.057998+0000","last_undegraded":"2024-04-15T15:17:00.057998+0000","last_fullsized":"2024-04-15T15:17:00.057998+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047704+0000","last_change":"2024-04-15T15:17:00.047704+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047704+0000","last_undegraded":"2024-04-15T15:17:00.047704+0000","last_fullsized":"2024-04-15T15:17:00.047704+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.055352+0000","last_change":"2024-04-15T15:17:00.052685+0000","last_active":"2024-04-15T15:17:00.055352+0000","last_peered":"2024-04-15T15:17:00.055352+0000","last_clean":"2024-04-15T15:17:00.055352+0000","last_became_active":"2024-04-15T15:17:00.052398+0000","last_became_peered":"2024-04-15T15:17:00.052398+0000","last_unstale":"2024-04-15T15:17:00.055352+0000","last_undegraded":"2024-04-15T15:17:00.055352+0000","last_fullsized":"2024-04-15T15:17:00.055352+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:42:40.378774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.048032+0000","last_change":"2024-04-15T15:17:00.048032+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.048032+0000","last_undegraded":"2024-04-15T15:17:00.048032+0000","last_fullsized":"2024-04-15T15:17:00.048032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.046519+0000","last_change":"2024-04-15T15:17:00.044538+0000","last_active":"2024-04-15T15:17:00.046519+0000","last_peered":"2024-04-15T15:17:00.046519+0000","last_clean":"2024-04-15T15:17:00.046519+0000","last_became_active":"2024-04-15T15:17:00.044237+0000","last_became_peered":"2024-04-15T15:17:00.044237+0000","last_unstale":"2024-04-15T15:17:00.046519+0000","last_undegraded":"2024-04-15T15:17:00.046519+0000","last_fullsized":"2024-04-15T15:17:00.046519+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:37:42.468987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073110+0000","last_change":"2024-04-15T15:17:01.066478+0000","last_active":"2024-04-15T15:17:01.073110+0000","last_peered":"2024-04-15T15:17:01.073110+0000","last_clean":"2024-04-15T15:17:01.073110+0000","last_became_active":"2024-04-15T15:17:01.066229+0000","last_became_peered":"2024-04-15T15:17:01.066229+0000","last_unstale":"2024-04-15T15:17:01.073110+0000","last_undegraded":"2024-04-15T15:17:01.073110+0000","last_fullsized":"2024-04-15T15:17:01.073110+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.066213+0000","last_change":"2024-04-15T15:17:00.064123+0000","last_active":"2024-04-15T15:17:00.066213+0000","last_peered":"2024-04-15T15:17:00.066213+0000","last_clean":"2024-04-15T15:17:00.066213+0000","last_became_active":"2024-04-15T15:17:00.063535+0000","last_became_peered":"2024-04-15T15:17:00.063535+0000","last_unstale":"2024-04-15T15:17:00.066213+0000","last_undegraded":"2024-04-15T15:17:00.066213+0000","last_fullsized":"2024-04-15T15:17:00.066213+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046719+0000","last_change":"2024-04-15T15:17:00.046719+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046719+0000","last_undegraded":"2024-04-15T15:17:00.046719+0000","last_fullsized":"2024-04-15T15:17:00.046719+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053705+0000","last_change":"2024-04-15T15:17:00.051113+0000","last_active":"2024-04-15T15:17:00.053705+0000","last_peered":"2024-04-15T15:17:00.053705+0000","last_clean":"2024-04-15T15:17:00.053705+0000","last_became_active":"2024-04-15T15:17:00.050824+0000","last_became_peered":"2024-04-15T15:17:00.050824+0000","last_unstale":"2024-04-15T15:17:00.053705+0000","last_undegraded":"2024-04-15T15:17:00.053705+0000","last_fullsized":"2024-04-15T15:17:00.053705+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:50:01.167747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043001+0000","last_change":"2024-04-15T15:17:00.043001+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043001+0000","last_undegraded":"2024-04-15T15:17:00.043001+0000","last_fullsized":"2024-04-15T15:17:00.043001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078424+0000","last_change":"2024-04-15T15:17:00.058333+0000","last_active":"2024-04-15T15:17:01.078424+0000","last_peered":"2024-04-15T15:17:01.078424+0000","last_clean":"2024-04-15T15:17:01.078424+0000","last_became_active":"2024-04-15T15:17:00.057892+0000","last_became_peered":"2024-04-15T15:17:00.057892+0000","last_unstale":"2024-04-15T15:17:01.078424+0000","last_undegraded":"2024-04-15T15:17:01.078424+0000","last_fullsized":"2024-04-15T15:17:01.078424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.052245+0000","last_change":"2024-04-15T15:17:00.052245+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.052245+0000","last_undegraded":"2024-04-15T15:17:00.052245+0000","last_fullsized":"2024-04-15T15:17:00.052245+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053028+0000","last_change":"2024-04-15T15:17:00.049576+0000","last_active":"2024-04-15T15:17:00.053028+0000","last_peered":"2024-04-15T15:17:00.053028+0000","last_clean":"2024-04-15T15:17:00.053028+0000","last_became_active":"2024-04-15T15:17:00.049280+0000","last_became_peered":"2024-04-15T15:17:00.049280+0000","last_unstale":"2024-04-15T15:17:00.053028+0000","last_undegraded":"2024-04-15T15:17:00.053028+0000","last_fullsized":"2024-04-15T15:17:00.053028+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:16:00.136473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046328+0000","last_change":"2024-04-15T15:17:00.046328+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046328+0000","last_undegraded":"2024-04-15T15:17:00.046328+0000","last_fullsized":"2024-04-15T15:17:00.046328+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.062211+0000","last_change":"2024-04-15T15:17:00.061061+0000","last_active":"2024-04-15T15:17:00.062211+0000","last_peered":"2024-04-15T15:17:00.062211+0000","last_clean":"2024-04-15T15:17:00.062211+0000","last_became_active":"2024-04-15T15:17:00.060731+0000","last_became_peered":"2024-04-15T15:17:00.060731+0000","last_unstale":"2024-04-15T15:17:00.062211+0000","last_undegraded":"2024-04-15T15:17:00.062211+0000","last_fullsized":"2024-04-15T15:17:00.062211+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.046258+0000","last_change":"2024-04-15T15:17:00.046258+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.046258+0000","last_undegraded":"2024-04-15T15:17:00.046258+0000","last_fullsized":"2024-04-15T15:17:00.046258+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064288+0000","last_change":"2024-04-15T15:17:00.063764+0000","last_active":"2024-04-15T15:17:00.064288+0000","last_peered":"2024-04-15T15:17:00.064288+0000","last_clean":"2024-04-15T15:17:00.064288+0000","last_became_active":"2024-04-15T15:17:00.063528+0000","last_became_peered":"2024-04-15T15:17:00.063528+0000","last_unstale":"2024-04-15T15:17:00.064288+0000","last_undegraded":"2024-04-15T15:17:00.064288+0000","last_fullsized":"2024-04-15T15:17:00.064288+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:04.551109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.043848+0000","last_change":"2024-04-15T15:17:00.043848+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.043848+0000","last_undegraded":"2024-04-15T15:17:00.043848+0000","last_fullsized":"2024-04-15T15:17:00.043848+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048642+0000","last_change":"2024-04-15T15:17:00.056627+0000","last_active":"2024-04-15T15:17:01.048642+0000","last_peered":"2024-04-15T15:17:01.048642+0000","last_clean":"2024-04-15T15:17:01.048642+0000","last_became_active":"2024-04-15T15:17:00.056317+0000","last_became_peered":"2024-04-15T15:17:00.056317+0000","last_unstale":"2024-04-15T15:17:01.048642+0000","last_undegraded":"2024-04-15T15:17:01.048642+0000","last_fullsized":"2024-04-15T15:17:01.048642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074263+0000","last_change":"2024-04-15T15:17:01.071745+0000","last_active":"2024-04-15T15:17:01.074263+0000","last_peered":"2024-04-15T15:17:01.074263+0000","last_clean":"2024-04-15T15:17:01.074263+0000","last_became_active":"2024-04-15T15:17:01.071466+0000","last_became_peered":"2024-04-15T15:17:01.071466+0000","last_unstale":"2024-04-15T15:17:01.074263+0000","last_undegraded":"2024-04-15T15:17:01.074263+0000","last_fullsized":"2024-04-15T15:17:01.074263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:04.108334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.054345+0000","last_change":"2024-04-15T15:17:00.057863+0000","last_active":"2024-04-15T15:17:01.054345+0000","last_peered":"2024-04-15T15:17:01.054345+0000","last_clean":"2024-04-15T15:17:01.054345+0000","last_became_active":"2024-04-15T15:17:00.057596+0000","last_became_peered":"2024-04-15T15:17:00.057596+0000","last_unstale":"2024-04-15T15:17:01.054345+0000","last_undegraded":"2024-04-15T15:17:01.054345+0000","last_fullsized":"2024-04-15T15:17:01.054345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":37,"state":"creating+peering","last_fresh":"2024-04-15T15:17:00.047469+0000","last_change":"2024-04-15T15:17:00.047469+0000","last_active":"2024-04-15T15:17:00.033889+0000","last_peered":"2024-04-15T15:17:00.033889+0000","last_clean":"2024-04-15T15:17:00.033889+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:17:00.047469+0000","last_undegraded":"2024-04-15T15:17:00.047469+0000","last_fullsized":"2024-04-15T15:17:00.047469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.060185+0000","last_change":"2024-04-15T15:17:00.058811+0000","last_active":"2024-04-15T15:17:00.060185+0000","last_peered":"2024-04-15T15:17:00.060185+0000","last_clean":"2024-04-15T15:17:00.060185+0000","last_became_active":"2024-04-15T15:17:00.058530+0000","last_became_peered":"2024-04-15T15:17:00.058530+0000","last_unstale":"2024-04-15T15:17:00.060185+0000","last_undegraded":"2024-04-15T15:17:00.060185+0000","last_fullsized":"2024-04-15T15:17:00.060185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:03.103510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074626+0000","last_change":"2024-04-15T15:17:01.070460+0000","last_active":"2024-04-15T15:17:01.074626+0000","last_peered":"2024-04-15T15:17:01.074626+0000","last_clean":"2024-04-15T15:17:01.074626+0000","last_became_active":"2024-04-15T15:17:01.070110+0000","last_became_peered":"2024-04-15T15:17:01.070110+0000","last_unstale":"2024-04-15T15:17:01.074626+0000","last_undegraded":"2024-04-15T15:17:01.074626+0000","last_fullsized":"2024-04-15T15:17:01.074626+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.064075+0000","last_change":"2024-04-15T15:17:00.061284+0000","last_active":"2024-04-15T15:17:00.064075+0000","last_peered":"2024-04-15T15:17:00.064075+0000","last_clean":"2024-04-15T15:17:00.064075+0000","last_became_active":"2024-04-15T15:17:00.060520+0000","last_became_peered":"2024-04-15T15:17:00.060520+0000","last_unstale":"2024-04-15T15:17:00.064075+0000","last_undegraded":"2024-04-15T15:17:00.064075+0000","last_fullsized":"2024-04-15T15:17:00.064075+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":198,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:17:00.053524+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:00.053524+0000","last_peered":"2024-04-15T15:17:00.053524+0000","last_clean":"2024-04-15T15:17:00.053524+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:00.053524+0000","last_undegraded":"2024-04-15T15:17:00.053524+0000","last_fullsized":"2024-04-15T15:17:00.053524+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":187741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1015808,"data_stored":770360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":179992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":173325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31592,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340248,"statfs":{"total":96636764160,"available":96604413952,"internally_reserved":0,"allocated":421888,"data_stored":179992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":1003520,"data_stored":763693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:05.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-04-15T15:17:05.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.048962+0000', 'last_change': '2024-04-15T15:17:00.048962+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.048962+0000', 'last_undegraded': '2024-04-15T15:17:00.048962+0000', 'last_fullsized': '2024-04-15T15:17:00.048962+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-04-15T15:17:05.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.046889+0000', 'last_change': '2024-04-15T15:17:00.046889+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.046889+0000', 'last_undegraded': '2024-04-15T15:17:00.046889+0000', 'last_fullsized': '2024-04-15T15:17:00.046889+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047885+0000', 'last_change': '2024-04-15T15:17:00.047885+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047885+0000', 'last_undegraded': '2024-04-15T15:17:00.047885+0000', 'last_fullsized': '2024-04-15T15:17:00.047885+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.045994+0000', 'last_change': '2024-04-15T15:17:00.045994+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.045994+0000', 'last_undegraded': '2024-04-15T15:17:00.045994+0000', 'last_fullsized': '2024-04-15T15:17:00.045994+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043764+0000', 'last_change': '2024-04-15T15:17:00.043764+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043764+0000', 'last_undegraded': '2024-04-15T15:17:00.043764+0000', 'last_fullsized': '2024-04-15T15:17:00.043764+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-04-15T15:17:05.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.044975+0000', 'last_change': '2024-04-15T15:17:00.044975+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.044975+0000', 'last_undegraded': '2024-04-15T15:17:00.044975+0000', 'last_fullsized': '2024-04-15T15:17:00.044975+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.049417+0000', 'last_change': '2024-04-15T15:17:00.049417+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.049417+0000', 'last_undegraded': '2024-04-15T15:17:00.049417+0000', 'last_fullsized': '2024-04-15T15:17:00.049417+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.049028+0000', 'last_change': '2024-04-15T15:17:00.049028+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.049028+0000', 'last_undegraded': '2024-04-15T15:17:00.049028+0000', 'last_fullsized': '2024-04-15T15:17:00.049028+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.045186+0000', 'last_change': '2024-04-15T15:17:00.045186+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.045186+0000', 'last_undegraded': '2024-04-15T15:17:00.045186+0000', 'last_fullsized': '2024-04-15T15:17:00.045186+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.052205+0000', 'last_change': '2024-04-15T15:17:00.052205+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.052205+0000', 'last_undegraded': '2024-04-15T15:17:00.052205+0000', 'last_fullsized': '2024-04-15T15:17:00.052205+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-15T15:17:05.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047966+0000', 'last_change': '2024-04-15T15:17:00.047966+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047966+0000', 'last_undegraded': '2024-04-15T15:17:00.047966+0000', 'last_fullsized': '2024-04-15T15:17:00.047966+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043158+0000', 'last_change': '2024-04-15T15:17:00.043158+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043158+0000', 'last_undegraded': '2024-04-15T15:17:00.043158+0000', 'last_fullsized': '2024-04-15T15:17:00.043158+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.052838+0000', 'last_change': '2024-04-15T15:17:00.052838+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.052838+0000', 'last_undegraded': '2024-04-15T15:17:00.052838+0000', 'last_fullsized': '2024-04-15T15:17:00.052838+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.042458+0000', 'last_change': '2024-04-15T15:17:00.042458+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.042458+0000', 'last_undegraded': '2024-04-15T15:17:00.042458+0000', 'last_fullsized': '2024-04-15T15:17:00.042458+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.044860+0000', 'last_change': '2024-04-15T15:17:00.044860+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.044860+0000', 'last_undegraded': '2024-04-15T15:17:00.044860+0000', 'last_fullsized': '2024-04-15T15:17:00.044860+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-15T15:17:05.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.048125+0000', 'last_change': '2024-04-15T15:17:00.048125+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.048125+0000', 'last_undegraded': '2024-04-15T15:17:00.048125+0000', 'last_fullsized': '2024-04-15T15:17:00.048125+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.044106+0000', 'last_change': '2024-04-15T15:17:00.044106+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.044106+0000', 'last_undegraded': '2024-04-15T15:17:00.044106+0000', 'last_fullsized': '2024-04-15T15:17:00.044106+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047283+0000', 'last_change': '2024-04-15T15:17:00.047283+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047283+0000', 'last_undegraded': '2024-04-15T15:17:00.047283+0000', 'last_fullsized': '2024-04-15T15:17:00.047283+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043633+0000', 'last_change': '2024-04-15T15:17:00.043633+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043633+0000', 'last_undegraded': '2024-04-15T15:17:00.043633+0000', 'last_fullsized': '2024-04-15T15:17:00.043633+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.050290+0000', 'last_change': '2024-04-15T15:17:00.050290+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.050290+0000', 'last_undegraded': '2024-04-15T15:17:00.050290+0000', 'last_fullsized': '2024-04-15T15:17:00.050290+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047108+0000', 'last_change': '2024-04-15T15:17:00.047108+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047108+0000', 'last_undegraded': '2024-04-15T15:17:00.047108+0000', 'last_fullsized': '2024-04-15T15:17:00.047108+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.050788+0000', 'last_change': '2024-04-15T15:17:00.050788+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.050788+0000', 'last_undegraded': '2024-04-15T15:17:00.050788+0000', 'last_fullsized': '2024-04-15T15:17:00.050788+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043571+0000', 'last_change': '2024-04-15T15:17:00.043571+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043571+0000', 'last_undegraded': '2024-04-15T15:17:00.043571+0000', 'last_fullsized': '2024-04-15T15:17:00.043571+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.052662+0000', 'last_change': '2024-04-15T15:17:00.052662+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.052662+0000', 'last_undegraded': '2024-04-15T15:17:00.052662+0000', 'last_fullsized': '2024-04-15T15:17:00.052662+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.045457+0000', 'last_change': '2024-04-15T15:17:00.045457+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.045457+0000', 'last_undegraded': '2024-04-15T15:17:00.045457+0000', 'last_fullsized': '2024-04-15T15:17:00.045457+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.046672+0000', 'last_change': '2024-04-15T15:17:00.046672+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.046672+0000', 'last_undegraded': '2024-04-15T15:17:00.046672+0000', 'last_fullsized': '2024-04-15T15:17:00.046672+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-04-15T15:17:05.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.048446+0000', 'last_change': '2024-04-15T15:17:00.048446+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.048446+0000', 'last_undegraded': '2024-04-15T15:17:00.048446+0000', 'last_fullsized': '2024-04-15T15:17:00.048446+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047149+0000', 'last_change': '2024-04-15T15:17:00.047149+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047149+0000', 'last_undegraded': '2024-04-15T15:17:00.047149+0000', 'last_fullsized': '2024-04-15T15:17:00.047149+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047661+0000', 'last_change': '2024-04-15T15:17:00.047661+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047661+0000', 'last_undegraded': '2024-04-15T15:17:00.047661+0000', 'last_fullsized': '2024-04-15T15:17:00.047661+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.051279+0000', 'last_change': '2024-04-15T15:17:00.051279+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.051279+0000', 'last_undegraded': '2024-04-15T15:17:00.051279+0000', 'last_fullsized': '2024-04-15T15:17:00.051279+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043001+0000', 'last_change': '2024-04-15T15:17:00.043001+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043001+0000', 'last_undegraded': '2024-04-15T15:17:00.043001+0000', 'last_fullsized': '2024-04-15T15:17:00.043001+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047704+0000', 'last_change': '2024-04-15T15:17:00.047704+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047704+0000', 'last_undegraded': '2024-04-15T15:17:00.047704+0000', 'last_fullsized': '2024-04-15T15:17:00.047704+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.048032+0000', 'last_change': '2024-04-15T15:17:00.048032+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.048032+0000', 'last_undegraded': '2024-04-15T15:17:00.048032+0000', 'last_fullsized': '2024-04-15T15:17:00.048032+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.046719+0000', 'last_change': '2024-04-15T15:17:00.046719+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.046719+0000', 'last_undegraded': '2024-04-15T15:17:00.046719+0000', 'last_fullsized': '2024-04-15T15:17:00.046719+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043001+0000', 'last_change': '2024-04-15T15:17:00.043001+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043001+0000', 'last_undegraded': '2024-04-15T15:17:00.043001+0000', 'last_fullsized': '2024-04-15T15:17:00.043001+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.052245+0000', 'last_change': '2024-04-15T15:17:00.052245+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.052245+0000', 'last_undegraded': '2024-04-15T15:17:00.052245+0000', 'last_fullsized': '2024-04-15T15:17:00.052245+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.046328+0000', 'last_change': '2024-04-15T15:17:00.046328+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.046328+0000', 'last_undegraded': '2024-04-15T15:17:00.046328+0000', 'last_fullsized': '2024-04-15T15:17:00.046328+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-04-15T15:17:05.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.046258+0000', 'last_change': '2024-04-15T15:17:00.046258+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.046258+0000', 'last_undegraded': '2024-04-15T15:17:00.046258+0000', 'last_fullsized': '2024-04-15T15:17:00.046258+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-15T15:17:05.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-04-15T15:17:05.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.043848+0000', 'last_change': '2024-04-15T15:17:00.043848+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.043848+0000', 'last_undegraded': '2024-04-15T15:17:00.043848+0000', 'last_fullsized': '2024-04-15T15:17:00.043848+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-15T15:17:05.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-15T15:17:05.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 37, 'state': 'creating+peering', 'last_fresh': '2024-04-15T15:17:00.047469+0000', 'last_change': '2024-04-15T15:17:00.047469+0000', 'last_active': '2024-04-15T15:17:00.033889+0000', 'last_peered': '2024-04-15T15:17:00.033889+0000', 'last_clean': '2024-04-15T15:17:00.033889+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:17:00.047469+0000', 'last_undegraded': '2024-04-15T15:17:00.047469+0000', 'last_fullsized': '2024-04-15T15:17:00.047469+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 37, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'last_clean_scrub_stamp': '2024-04-15T15:17:00.033889+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-15T15:17:05.825 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-15T15:17:05.848 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-04-15T15:17:05.846+0000 7fa308937640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:17:05.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.949+0000 7f8c473cd640 1 Processor -- start 2024-04-15T15:17:05.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.949+0000 7f8c473cd640 1 -- start start 2024-04-15T15:17:05.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c473cd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4009a9d0 0x7f8c4009adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c473cd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c400d02b0 0x7f8c4009b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c473cd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4015cc50 con 0x7f8c40160340 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c473cd640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8c4009b2f0 con 0x7f8c4009a9d0 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c473cd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4015cee0 con 0x7f8c400d02b0 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c45169640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8c4009a9d0 0x7f8c4009adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c4596a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c40160340 0x7f8c401607d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52256/0 (socket says 172.21.15.46:52256) 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.950+0000 7f8c4596a640 1 -- 172.21.15.46:0/1482030764 learned_addr learned my addr 172.21.15.46:0/1482030764 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c44968640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c400d02b0 0x7f8c4009b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c45169640 1 -- 172.21.15.46:0/1482030764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c400d02b0 msgr2=0x7f8c4009b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c45169640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c400d02b0 0x7f8c4009b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c45169640 1 -- 172.21.15.46:0/1482030764 >> v1:172.21.15.46:6789/0 conn(0x7f8c40160340 legacy=0x7f8c401607d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c45169640 1 -- 172.21.15.46:0/1482030764 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4015d1c0 con 0x7f8c4009a9d0 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c3ffff640 1 -- 172.21.15.46:0/1482030764 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963218295 0 0) 0x7f8c4015cc50 con 0x7f8c40160340 2024-04-15T15:17:05.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.951+0000 7f8c44968640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c400d02b0 0x7f8c4009b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c45169640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4009a9d0 0x7f8c4009adb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009d90 tx=0x7f8c3002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b65dd22ebceef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c3ffff640 1 -- 172.21.15.46:0/1482030764 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30039070 con 0x7f8c4009a9d0 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c3ffff640 1 -- 172.21.15.46:0/1482030764 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3002cc50 con 0x7f8c4009a9d0 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c3ffff640 1 -- 172.21.15.46:0/1482030764 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30034440 con 0x7f8c4009a9d0 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 -- 172.21.15.46:0/1482030764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4009a9d0 msgr2=0x7f8c4009adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4009a9d0 0x7f8c4009adb0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009d90 tx=0x7f8c3002bea0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 -- 172.21.15.46:0/1482030764 shutdown_connections 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c400d02b0 0x7f8c4009b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 --2- 172.21.15.46:0/1482030764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c4009a9d0 0x7f8c4009adb0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 -- 172.21.15.46:0/1482030764 >> 172.21.15.46:0/1482030764 conn(0x7f8c400b3e70 msgr2=0x7f8c400b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 -- 172.21.15.46:0/1482030764 shutdown_connections 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.952+0000 7f8c473cd640 1 -- 172.21.15.46:0/1482030764 wait complete. 2024-04-15T15:17:05.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 Processor -- start 2024-04-15T15:17:05.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 -- start start 2024-04-15T15:17:05.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4009a9d0 0x7f8c40152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c400d02b0 0x7f8c400b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 0x7f8c4015f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4015da60 con 0x7f8c40160340 2024-04-15T15:17:05.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4015d780 con 0x7f8c400d02b0 2024-04-15T15:17:05.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c473cd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4009bc60 con 0x7f8c4009a9d0 2024-04-15T15:17:05.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c4596a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 0x7f8c4015f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c4596a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 0x7f8c4015f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48188/0 (socket says 172.21.15.46:48188) 2024-04-15T15:17:05.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.954+0000 7f8c4596a640 1 -- 172.21.15.46:0/910448484 learned_addr learned my addr 172.21.15.46:0/910448484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:05.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c45169640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4009a9d0 0x7f8c40152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c44968640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c400d02b0 0x7f8c400b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c4596a640 1 -- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4009a9d0 msgr2=0x7f8c40152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c4596a640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4009a9d0 0x7f8c40152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c4596a640 1 -- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c400d02b0 msgr2=0x7f8c400b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c4596a640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c400d02b0 0x7f8c400b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c4596a640 1 -- 172.21.15.46:0/910448484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c400b6850 con 0x7f8c40160340 2024-04-15T15:17:05.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c45169640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4009a9d0 0x7f8c40152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:05.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c4596a640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 0x7f8c4015f460 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c3e0 tx=0x7f8c3800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c380074e0 con 0x7f8c40160340 2024-04-15T15:17:05.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c380047e0 con 0x7f8c40160340 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38004b90 con 0x7f8c40160340 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c400b6b30 con 0x7f8c40160340 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.955+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c400b70f0 con 0x7f8c40160340 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.956+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c400690f0 con 0x7f8c40160340 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.957+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c38011070 con 0x7f8c40160340 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.957+0000 7f8c3dffb640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.957+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8c3808b910 con 0x7f8c40160340 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.957+0000 7f8c45169640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.958+0000 7f8c45169640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8c30036000 tx=0x7f8c30030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:05.961+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c3800e280 con 0x7f8c40160340 2024-04-15T15:17:06.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.218+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8c400b8240 con 0x7f8c40160340 2024-04-15T15:17:06.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.218+0000 7f8c3dffb640 1 -- 172.21.15.46:0/910448484 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1098 (secure 0 0 0) 0x7f8c38059bb0 con 0x7f8c40160340 2024-04-15T15:17:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:06.220 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1713194125,"num_in_osds":8,"osd_in_since":1713194116,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":57},{"state_name":"creating+peering","count":40}],"num_pgs":97,"num_pools":3,"num_objects":10,"data_bytes":591062,"bytes_used":242999296,"bytes_avail":772851113984,"bytes_total":773094113280,"inactive_pgs_ratio":0.41237112879753113,"write_bytes_sec":575,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":17,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5454}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-15T15:16:44.368753+0000","services":{}},"progress_events":{}} 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 msgr2=0x7f8c2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8c30036000 tx=0x7f8c30030000 comp rx=0 tx=0).stop 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 msgr2=0x7f8c4015f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 0x7f8c4015f460 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800c3e0 tx=0x7f8c3800c8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 shutdown_connections 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8c2c06c8f0 0x7f8c2c06edb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40160340 0x7f8c4015f460 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c400d02b0 0x7f8c400b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 --2- 172.21.15.46:0/910448484 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8c4009a9d0 0x7f8c40152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 >> 172.21.15.46:0/910448484 conn(0x7f8c400b3e70 msgr2=0x7f8c400b4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 shutdown_connections 2024-04-15T15:17:06.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.220+0000 7f8c473cd640 1 -- 172.21.15.46:0/910448484 wait complete. 2024-04-15T15:17:06.232 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-15T15:17:06.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.335+0000 7f9fc8fba640 1 Processor -- start 2024-04-15T15:17:06.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.335+0000 7f9fc8fba640 1 -- start start 2024-04-15T15:17:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc8fba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fc4070940 0x7f9fc409b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc8fba640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f9fc4160310 0x7f9fc4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc8fba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc40d0da0 0x7f9fc40d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc8fba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f9fc40d05e0 con 0x7f9fc4070940 2024-04-15T15:17:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc8fba640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f9fc4071260 con 0x7f9fc4160310 2024-04-15T15:17:06.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc8fba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9fc415f720 con 0x7f9fc40d0da0 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fc4070940 0x7f9fc409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9fc4070940 0x7f9fc409b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48192/0 (socket says 172.21.15.46:48192) 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.336+0000 7f9fc259c640 1 -- 172.21.15.46:0/650750043 learned_addr learned my addr 172.21.15.46:0/650750043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc2d9d640 1 --2- 172.21.15.46:0/650750043 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc40d0da0 0x7f9fc40d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc1d9b640 1 --2- 172.21.15.46:0/650750043 >> v2:172.21.15.78:3300/0 conn(0x7f9fc4160310 0x7f9fc4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc259c640 1 -- 172.21.15.46:0/650750043 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc40d0da0 msgr2=0x7f9fc40d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc259c640 1 --2- 172.21.15.46:0/650750043 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc40d0da0 0x7f9fc40d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc259c640 1 -- 172.21.15.46:0/650750043 >> v2:172.21.15.78:3300/0 conn(0x7f9fc4160310 msgr2=0x7f9fc4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc259c640 1 --2- 172.21.15.46:0/650750043 >> v2:172.21.15.78:3300/0 conn(0x7f9fc4160310 0x7f9fc4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc259c640 1 -- 172.21.15.46:0/650750043 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc415fa00 con 0x7f9fc4070940 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc1d9b640 1 --2- 172.21.15.46:0/650750043 >> v2:172.21.15.78:3300/0 conn(0x7f9fc4160310 0x7f9fc4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:06.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.337+0000 7f9fc259c640 1 --2- 172.21.15.46:0/650750043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4070940 0x7f9fc409b880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40734f0 tx=0x7f9fb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89333752de5d194 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc159a640 1 -- 172.21.15.46:0/650750043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb402ca70 con 0x7f9fc4070940 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc159a640 1 -- 172.21.15.46:0/650750043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fb402cc10 con 0x7f9fc4070940 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc159a640 1 -- 172.21.15.46:0/650750043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40343f0 con 0x7f9fc4070940 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 -- 172.21.15.46:0/650750043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4070940 msgr2=0x7f9fc409b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/650750043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4070940 0x7f9fc409b880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40734f0 tx=0x7f9fb402bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 -- 172.21.15.46:0/650750043 shutdown_connections 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/650750043 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc40d0da0 0x7f9fc40d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/650750043 >> v2:172.21.15.78:3300/0 conn(0x7f9fc4160310 0x7f9fc4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/650750043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4070940 0x7f9fc409b880 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 -- 172.21.15.46:0/650750043 >> 172.21.15.46:0/650750043 conn(0x7f9fc40b3f40 msgr2=0x7f9fc40b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 -- 172.21.15.46:0/650750043 shutdown_connections 2024-04-15T15:17:06.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.338+0000 7f9fc8fba640 1 -- 172.21.15.46:0/650750043 wait complete. 2024-04-15T15:17:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 Processor -- start 2024-04-15T15:17:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 -- start start 2024-04-15T15:17:06.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc4070940 0x7f9fc409e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc40d0da0 0x7f9fc40c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 0x7f9fc40c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc4160770 con 0x7f9fc4160310 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc40d0880 con 0x7f9fc40d0da0 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc8fba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9fc40d1750 con 0x7f9fc4070940 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.340+0000 7f9fc2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 0x7f9fc40c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 0x7f9fc40c7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48206/0 (socket says 172.21.15.46:48206) 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 -- 172.21.15.46:0/3383074649 learned_addr learned my addr 172.21.15.46:0/3383074649 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc259c640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc4070940 0x7f9fc409e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc1d9b640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc40d0da0 0x7f9fc40c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 -- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc4070940 msgr2=0x7f9fc409e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc4070940 0x7f9fc409e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 -- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc40d0da0 msgr2=0x7f9fc40c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc40d0da0 0x7f9fc40c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 -- 172.21.15.46:0/3383074649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc40c5f60 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc2d9d640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 0x7f9fc40c7540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00d830 tx=0x7f9fbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc018020 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fbc0096c0 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fc40c6240 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc009a90 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.341+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fc40c6800 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.342+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fc40689e0 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.343+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9fbc01b020 con 0x7f9fc4160310 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.343+0000 7f9fbaffd640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9fa406c8f0 0x7f9fa406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.343+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9fbc08c0a0 con 0x7f9fc4160310 2024-04-15T15:17:06.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.344+0000 7f9fc259c640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9fa406c8f0 0x7f9fa406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.344+0000 7f9fc259c640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9fa406c8f0 0x7f9fa406edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4035000 tx=0x7f9fb4030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.348+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fbc022030 con 0x7f9fc4160310 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.533+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9fc4070720 con 0x7f9fa406c8f0 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.539+0000 7f9fbaffd640 1 -- 172.21.15.46:0/3383074649 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251933 (secure 0 0 0) 0x7f9fc4070720 con 0x7f9fa406c8f0 2024-04-15T15:17:06.545 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:06.546 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9fa406c8f0 msgr2=0x7f9fa406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9fa406c8f0 0x7f9fa406edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4035000 tx=0x7f9fb4030000 comp rx=0 tx=0).stop 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 msgr2=0x7f9fc40c7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 0x7f9fc40c7540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00d830 tx=0x7f9fbc00dcf0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 shutdown_connections 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9fa406c8f0 0x7f9fa406edb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fc4160310 0x7f9fc40c7540 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fc40d0da0 0x7f9fc40c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 --2- 172.21.15.46:0/3383074649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9fc4070940 0x7f9fc409e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 >> 172.21.15.46:0/3383074649 conn(0x7f9fc40b3f40 msgr2=0x7f9fc40b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 shutdown_connections 2024-04-15T15:17:06.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:06.542+0000 7f9fc8fba640 1 -- 172.21.15.46:0/3383074649 wait complete. 2024-04-15T15:17:06.555 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-15T15:17:06.373621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237512,"kb_used_data":4676,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737208,"statfs":{"total":773094113280,"available":772850900992,"internally_reserved":0,"allocated":4788224,"data_stored":2716288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.335358"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378116+0000","last_change":"2024-04-15T15:17:01.062449+0000","last_active":"2024-04-15T15:17:02.378116+0000","last_peered":"2024-04-15T15:17:02.378116+0000","last_clean":"2024-04-15T15:17:02.378116+0000","last_became_active":"2024-04-15T15:17:01.060871+0000","last_became_peered":"2024-04-15T15:17:01.060871+0000","last_unstale":"2024-04-15T15:17:02.378116+0000","last_undegraded":"2024-04-15T15:17:02.378116+0000","last_fullsized":"2024-04-15T15:17:02.378116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:44.232483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378008+0000","last_change":"2024-04-15T15:17:01.058267+0000","last_active":"2024-04-15T15:17:02.378008+0000","last_peered":"2024-04-15T15:17:02.378008+0000","last_clean":"2024-04-15T15:17:02.378008+0000","last_became_active":"2024-04-15T15:17:01.057296+0000","last_became_peered":"2024-04-15T15:17:01.057296+0000","last_unstale":"2024-04-15T15:17:02.378008+0000","last_undegraded":"2024-04-15T15:17:02.378008+0000","last_fullsized":"2024-04-15T15:17:02.378008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:07:59.866764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377486+0000","last_change":"2024-04-15T15:17:01.067156+0000","last_active":"2024-04-15T15:17:02.377486+0000","last_peered":"2024-04-15T15:17:02.377486+0000","last_clean":"2024-04-15T15:17:02.377486+0000","last_became_active":"2024-04-15T15:17:01.066771+0000","last_became_peered":"2024-04-15T15:17:01.066771+0000","last_unstale":"2024-04-15T15:17:02.377486+0000","last_undegraded":"2024-04-15T15:17:02.377486+0000","last_fullsized":"2024-04-15T15:17:02.377486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075545+0000","last_change":"2024-04-15T15:17:01.071566+0000","last_active":"2024-04-15T15:17:02.075545+0000","last_peered":"2024-04-15T15:17:02.075545+0000","last_clean":"2024-04-15T15:17:02.075545+0000","last_became_active":"2024-04-15T15:17:01.071282+0000","last_became_peered":"2024-04-15T15:17:01.071282+0000","last_unstale":"2024-04-15T15:17:02.075545+0000","last_undegraded":"2024-04-15T15:17:02.075545+0000","last_fullsized":"2024-04-15T15:17:02.075545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075285+0000","last_change":"2024-04-15T15:17:01.071320+0000","last_active":"2024-04-15T15:17:02.075285+0000","last_peered":"2024-04-15T15:17:02.075285+0000","last_clean":"2024-04-15T15:17:02.075285+0000","last_became_active":"2024-04-15T15:17:01.071021+0000","last_became_peered":"2024-04-15T15:17:01.071021+0000","last_unstale":"2024-04-15T15:17:02.075285+0000","last_undegraded":"2024-04-15T15:17:02.075285+0000","last_fullsized":"2024-04-15T15:17:02.075285+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:59.380296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060471+0000","last_change":"2024-04-15T15:17:01.064800+0000","last_active":"2024-04-15T15:17:02.060471+0000","last_peered":"2024-04-15T15:17:02.060471+0000","last_clean":"2024-04-15T15:17:02.060471+0000","last_became_active":"2024-04-15T15:17:01.064425+0000","last_became_peered":"2024-04-15T15:17:01.064425+0000","last_unstale":"2024-04-15T15:17:02.060471+0000","last_undegraded":"2024-04-15T15:17:02.060471+0000","last_fullsized":"2024-04-15T15:17:02.060471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074681+0000","last_change":"2024-04-15T15:17:01.069002+0000","last_active":"2024-04-15T15:17:01.074681+0000","last_peered":"2024-04-15T15:17:01.074681+0000","last_clean":"2024-04-15T15:17:01.074681+0000","last_became_active":"2024-04-15T15:17:01.068755+0000","last_became_peered":"2024-04-15T15:17:01.068755+0000","last_unstale":"2024-04-15T15:17:01.074681+0000","last_undegraded":"2024-04-15T15:17:01.074681+0000","last_fullsized":"2024-04-15T15:17:01.074681+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377473+0000","last_change":"2024-04-15T15:17:01.071878+0000","last_active":"2024-04-15T15:17:02.377473+0000","last_peered":"2024-04-15T15:17:02.377473+0000","last_clean":"2024-04-15T15:17:02.377473+0000","last_became_active":"2024-04-15T15:17:01.070505+0000","last_became_peered":"2024-04-15T15:17:01.070505+0000","last_unstale":"2024-04-15T15:17:02.377473+0000","last_undegraded":"2024-04-15T15:17:02.377473+0000","last_fullsized":"2024-04-15T15:17:02.377473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:25.547677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378040+0000","last_change":"2024-04-15T15:17:01.066415+0000","last_active":"2024-04-15T15:17:02.378040+0000","last_peered":"2024-04-15T15:17:02.378040+0000","last_clean":"2024-04-15T15:17:02.378040+0000","last_became_active":"2024-04-15T15:17:01.066137+0000","last_became_peered":"2024-04-15T15:17:01.066137+0000","last_unstale":"2024-04-15T15:17:02.378040+0000","last_undegraded":"2024-04-15T15:17:02.378040+0000","last_fullsized":"2024-04-15T15:17:02.378040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073089+0000","last_change":"2024-04-15T15:17:01.068258+0000","last_active":"2024-04-15T15:17:01.073089+0000","last_peered":"2024-04-15T15:17:01.073089+0000","last_clean":"2024-04-15T15:17:01.073089+0000","last_became_active":"2024-04-15T15:17:01.067888+0000","last_became_peered":"2024-04-15T15:17:01.067888+0000","last_unstale":"2024-04-15T15:17:01.073089+0000","last_undegraded":"2024-04-15T15:17:01.073089+0000","last_fullsized":"2024-04-15T15:17:01.073089+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075397+0000","last_change":"2024-04-15T15:17:01.070976+0000","last_active":"2024-04-15T15:17:01.075397+0000","last_peered":"2024-04-15T15:17:01.075397+0000","last_clean":"2024-04-15T15:17:01.075397+0000","last_became_active":"2024-04-15T15:17:01.070678+0000","last_became_peered":"2024-04-15T15:17:01.070678+0000","last_unstale":"2024-04-15T15:17:01.075397+0000","last_undegraded":"2024-04-15T15:17:01.075397+0000","last_fullsized":"2024-04-15T15:17:01.075397+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:38:18.560062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377736+0000","last_change":"2024-04-15T15:17:01.056955+0000","last_active":"2024-04-15T15:17:02.377736+0000","last_peered":"2024-04-15T15:17:02.377736+0000","last_clean":"2024-04-15T15:17:02.377736+0000","last_became_active":"2024-04-15T15:17:01.056604+0000","last_became_peered":"2024-04-15T15:17:01.056604+0000","last_unstale":"2024-04-15T15:17:02.377736+0000","last_undegraded":"2024-04-15T15:17:02.377736+0000","last_fullsized":"2024-04-15T15:17:02.377736+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.065827+0000","last_change":"2024-04-15T15:17:01.064962+0000","last_active":"2024-04-15T15:17:01.065827+0000","last_peered":"2024-04-15T15:17:01.065827+0000","last_clean":"2024-04-15T15:17:01.065827+0000","last_became_active":"2024-04-15T15:17:01.064395+0000","last_became_peered":"2024-04-15T15:17:01.064395+0000","last_unstale":"2024-04-15T15:17:01.065827+0000","last_undegraded":"2024-04-15T15:17:01.065827+0000","last_fullsized":"2024-04-15T15:17:01.065827+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072393+0000","last_change":"2024-04-15T15:17:01.069059+0000","last_active":"2024-04-15T15:17:01.072393+0000","last_peered":"2024-04-15T15:17:01.072393+0000","last_clean":"2024-04-15T15:17:01.072393+0000","last_became_active":"2024-04-15T15:17:01.067835+0000","last_became_peered":"2024-04-15T15:17:01.067835+0000","last_unstale":"2024-04-15T15:17:01.072393+0000","last_undegraded":"2024-04-15T15:17:01.072393+0000","last_fullsized":"2024-04-15T15:17:01.072393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073052+0000","last_change":"2024-04-15T15:17:01.065939+0000","last_active":"2024-04-15T15:17:01.073052+0000","last_peered":"2024-04-15T15:17:01.073052+0000","last_clean":"2024-04-15T15:17:01.073052+0000","last_became_active":"2024-04-15T15:17:01.065641+0000","last_became_peered":"2024-04-15T15:17:01.065641+0000","last_unstale":"2024-04-15T15:17:01.073052+0000","last_undegraded":"2024-04-15T15:17:01.073052+0000","last_fullsized":"2024-04-15T15:17:01.073052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377846+0000","last_change":"2024-04-15T15:17:01.064428+0000","last_active":"2024-04-15T15:17:02.377846+0000","last_peered":"2024-04-15T15:17:02.377846+0000","last_clean":"2024-04-15T15:17:02.377846+0000","last_became_active":"2024-04-15T15:17:01.064147+0000","last_became_peered":"2024-04-15T15:17:01.064147+0000","last_unstale":"2024-04-15T15:17:02.377846+0000","last_undegraded":"2024-04-15T15:17:02.377846+0000","last_fullsized":"2024-04-15T15:17:02.377846+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378393+0000","last_change":"2024-04-15T15:17:01.067149+0000","last_active":"2024-04-15T15:17:02.378393+0000","last_peered":"2024-04-15T15:17:02.378393+0000","last_clean":"2024-04-15T15:17:02.378393+0000","last_became_active":"2024-04-15T15:17:01.066871+0000","last_became_peered":"2024-04-15T15:17:01.066871+0000","last_unstale":"2024-04-15T15:17:02.378393+0000","last_undegraded":"2024-04-15T15:17:02.378393+0000","last_fullsized":"2024-04-15T15:17:02.378393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:24.305106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073211+0000","last_change":"2024-04-15T15:17:01.070470+0000","last_active":"2024-04-15T15:17:01.073211+0000","last_peered":"2024-04-15T15:17:01.073211+0000","last_clean":"2024-04-15T15:17:01.073211+0000","last_became_active":"2024-04-15T15:17:01.070200+0000","last_became_peered":"2024-04-15T15:17:01.070200+0000","last_unstale":"2024-04-15T15:17:01.073211+0000","last_undegraded":"2024-04-15T15:17:01.073211+0000","last_fullsized":"2024-04-15T15:17:01.073211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075348+0000","last_change":"2024-04-15T15:17:01.057828+0000","last_active":"2024-04-15T15:17:02.075348+0000","last_peered":"2024-04-15T15:17:02.075348+0000","last_clean":"2024-04-15T15:17:02.075348+0000","last_became_active":"2024-04-15T15:17:01.057532+0000","last_became_peered":"2024-04-15T15:17:01.057532+0000","last_unstale":"2024-04-15T15:17:02.075348+0000","last_undegraded":"2024-04-15T15:17:02.075348+0000","last_fullsized":"2024-04-15T15:17:02.075348+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.076558+0000","last_change":"2024-04-15T15:17:01.071687+0000","last_active":"2024-04-15T15:17:01.076558+0000","last_peered":"2024-04-15T15:17:01.076558+0000","last_clean":"2024-04-15T15:17:01.076558+0000","last_became_active":"2024-04-15T15:17:01.071330+0000","last_became_peered":"2024-04-15T15:17:01.071330+0000","last_unstale":"2024-04-15T15:17:01.076558+0000","last_undegraded":"2024-04-15T15:17:01.076558+0000","last_fullsized":"2024-04-15T15:17:01.076558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:24.181074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073107+0000","last_change":"2024-04-15T15:17:01.067759+0000","last_active":"2024-04-15T15:17:01.073107+0000","last_peered":"2024-04-15T15:17:01.073107+0000","last_clean":"2024-04-15T15:17:01.073107+0000","last_became_active":"2024-04-15T15:17:01.067476+0000","last_became_peered":"2024-04-15T15:17:01.067476+0000","last_unstale":"2024-04-15T15:17:01.073107+0000","last_undegraded":"2024-04-15T15:17:01.073107+0000","last_fullsized":"2024-04-15T15:17:01.073107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072415+0000","last_change":"2024-04-15T15:17:01.070298+0000","last_active":"2024-04-15T15:17:01.072415+0000","last_peered":"2024-04-15T15:17:01.072415+0000","last_clean":"2024-04-15T15:17:01.072415+0000","last_became_active":"2024-04-15T15:17:01.070032+0000","last_became_peered":"2024-04-15T15:17:01.070032+0000","last_unstale":"2024-04-15T15:17:01.072415+0000","last_undegraded":"2024-04-15T15:17:01.072415+0000","last_fullsized":"2024-04-15T15:17:01.072415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:23.034253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377955+0000","last_change":"2024-04-15T15:17:01.058285+0000","last_active":"2024-04-15T15:17:02.377955+0000","last_peered":"2024-04-15T15:17:02.377955+0000","last_clean":"2024-04-15T15:17:02.377955+0000","last_became_active":"2024-04-15T15:17:01.058001+0000","last_became_peered":"2024-04-15T15:17:01.058001+0000","last_unstale":"2024-04-15T15:17:02.377955+0000","last_undegraded":"2024-04-15T15:17:02.377955+0000","last_fullsized":"2024-04-15T15:17:02.377955+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075300+0000","last_change":"2024-04-15T15:17:01.064638+0000","last_active":"2024-04-15T15:17:02.075300+0000","last_peered":"2024-04-15T15:17:02.075300+0000","last_clean":"2024-04-15T15:17:02.075300+0000","last_became_active":"2024-04-15T15:17:01.064289+0000","last_became_peered":"2024-04-15T15:17:01.064289+0000","last_unstale":"2024-04-15T15:17:02.075300+0000","last_undegraded":"2024-04-15T15:17:02.075300+0000","last_fullsized":"2024-04-15T15:17:02.075300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:51.299505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061210+0000","last_change":"2024-04-15T15:17:01.056648+0000","last_active":"2024-04-15T15:17:01.061210+0000","last_peered":"2024-04-15T15:17:01.061210+0000","last_clean":"2024-04-15T15:17:01.061210+0000","last_became_active":"2024-04-15T15:17:01.056354+0000","last_became_peered":"2024-04-15T15:17:01.056354+0000","last_unstale":"2024-04-15T15:17:01.061210+0000","last_undegraded":"2024-04-15T15:17:01.061210+0000","last_fullsized":"2024-04-15T15:17:01.061210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060738+0000","last_change":"2024-04-15T15:17:01.067781+0000","last_active":"2024-04-15T15:17:02.060738+0000","last_peered":"2024-04-15T15:17:02.060738+0000","last_clean":"2024-04-15T15:17:02.060738+0000","last_became_active":"2024-04-15T15:17:01.067517+0000","last_became_peered":"2024-04-15T15:17:01.067517+0000","last_unstale":"2024-04-15T15:17:02.060738+0000","last_undegraded":"2024-04-15T15:17:02.060738+0000","last_fullsized":"2024-04-15T15:17:02.060738+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.062082+0000","last_change":"2024-04-15T15:17:01.059015+0000","last_active":"2024-04-15T15:17:01.062082+0000","last_peered":"2024-04-15T15:17:01.062082+0000","last_clean":"2024-04-15T15:17:01.062082+0000","last_became_active":"2024-04-15T15:17:01.058718+0000","last_became_peered":"2024-04-15T15:17:01.058718+0000","last_unstale":"2024-04-15T15:17:01.062082+0000","last_undegraded":"2024-04-15T15:17:01.062082+0000","last_fullsized":"2024-04-15T15:17:01.062082+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074673+0000","last_change":"2024-04-15T15:17:01.069694+0000","last_active":"2024-04-15T15:17:01.074673+0000","last_peered":"2024-04-15T15:17:01.074673+0000","last_clean":"2024-04-15T15:17:01.074673+0000","last_became_active":"2024-04-15T15:17:01.069408+0000","last_became_peered":"2024-04-15T15:17:01.069408+0000","last_unstale":"2024-04-15T15:17:01.074673+0000","last_undegraded":"2024-04-15T15:17:01.074673+0000","last_fullsized":"2024-04-15T15:17:01.074673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072620+0000","last_change":"2024-04-15T15:17:01.068919+0000","last_active":"2024-04-15T15:17:01.072620+0000","last_peered":"2024-04-15T15:17:01.072620+0000","last_clean":"2024-04-15T15:17:01.072620+0000","last_became_active":"2024-04-15T15:17:01.068639+0000","last_became_peered":"2024-04-15T15:17:01.068639+0000","last_unstale":"2024-04-15T15:17:01.072620+0000","last_undegraded":"2024-04-15T15:17:01.072620+0000","last_fullsized":"2024-04-15T15:17:01.072620+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.061418+0000","last_change":"2024-04-15T15:17:01.056050+0000","last_active":"2024-04-15T15:17:01.061418+0000","last_peered":"2024-04-15T15:17:01.061418+0000","last_clean":"2024-04-15T15:17:01.061418+0000","last_became_active":"2024-04-15T15:17:01.055735+0000","last_became_peered":"2024-04-15T15:17:01.055735+0000","last_unstale":"2024-04-15T15:17:01.061418+0000","last_undegraded":"2024-04-15T15:17:01.061418+0000","last_fullsized":"2024-04-15T15:17:01.061418+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:51.299505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378118+0000","last_change":"2024-04-15T15:17:01.058787+0000","last_active":"2024-04-15T15:17:02.378118+0000","last_peered":"2024-04-15T15:17:02.378118+0000","last_clean":"2024-04-15T15:17:02.378118+0000","last_became_active":"2024-04-15T15:17:01.058483+0000","last_became_peered":"2024-04-15T15:17:01.058483+0000","last_unstale":"2024-04-15T15:17:02.378118+0000","last_undegraded":"2024-04-15T15:17:02.378118+0000","last_fullsized":"2024-04-15T15:17:02.378118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:38:18.560062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060461+0000","last_change":"2024-04-15T15:17:01.069926+0000","last_active":"2024-04-15T15:17:02.060461+0000","last_peered":"2024-04-15T15:17:02.060461+0000","last_clean":"2024-04-15T15:17:02.060461+0000","last_became_active":"2024-04-15T15:17:01.069632+0000","last_became_peered":"2024-04-15T15:17:01.069632+0000","last_unstale":"2024-04-15T15:17:02.060461+0000","last_undegraded":"2024-04-15T15:17:02.060461+0000","last_fullsized":"2024-04-15T15:17:02.060461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378199+0000","last_change":"2024-04-15T15:17:01.064698+0000","last_active":"2024-04-15T15:17:02.378199+0000","last_peered":"2024-04-15T15:17:02.378199+0000","last_clean":"2024-04-15T15:17:02.378199+0000","last_became_active":"2024-04-15T15:17:01.064241+0000","last_became_peered":"2024-04-15T15:17:01.064241+0000","last_unstale":"2024-04-15T15:17:02.378199+0000","last_undegraded":"2024-04-15T15:17:02.378199+0000","last_fullsized":"2024-04-15T15:17:02.378199+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:25.547677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078586+0000","last_change":"2024-04-15T15:17:00.053485+0000","last_active":"2024-04-15T15:17:01.078586+0000","last_peered":"2024-04-15T15:17:01.078586+0000","last_clean":"2024-04-15T15:17:01.078586+0000","last_became_active":"2024-04-15T15:17:00.053211+0000","last_became_peered":"2024-04-15T15:17:00.053211+0000","last_unstale":"2024-04-15T15:17:01.078586+0000","last_undegraded":"2024-04-15T15:17:01.078586+0000","last_fullsized":"2024-04-15T15:17:01.078586+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060777+0000","last_change":"2024-04-15T15:17:01.066600+0000","last_active":"2024-04-15T15:17:02.060777+0000","last_peered":"2024-04-15T15:17:02.060777+0000","last_clean":"2024-04-15T15:17:02.060777+0000","last_became_active":"2024-04-15T15:17:01.066291+0000","last_became_peered":"2024-04-15T15:17:01.066291+0000","last_unstale":"2024-04-15T15:17:02.060777+0000","last_undegraded":"2024-04-15T15:17:02.060777+0000","last_fullsized":"2024-04-15T15:17:02.060777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377872+0000","last_change":"2024-04-15T15:17:00.056209+0000","last_active":"2024-04-15T15:17:02.377872+0000","last_peered":"2024-04-15T15:17:02.377872+0000","last_clean":"2024-04-15T15:17:02.377872+0000","last_became_active":"2024-04-15T15:17:00.055881+0000","last_became_peered":"2024-04-15T15:17:00.055881+0000","last_unstale":"2024-04-15T15:17:02.377872+0000","last_undegraded":"2024-04-15T15:17:02.377872+0000","last_fullsized":"2024-04-15T15:17:02.377872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072065+0000","last_change":"2024-04-15T15:17:01.067226+0000","last_active":"2024-04-15T15:17:01.072065+0000","last_peered":"2024-04-15T15:17:01.072065+0000","last_clean":"2024-04-15T15:17:01.072065+0000","last_became_active":"2024-04-15T15:17:01.066479+0000","last_became_peered":"2024-04-15T15:17:01.066479+0000","last_unstale":"2024-04-15T15:17:01.072065+0000","last_undegraded":"2024-04-15T15:17:01.072065+0000","last_fullsized":"2024-04-15T15:17:01.072065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060829+0000","last_change":"2024-04-15T15:17:00.054251+0000","last_active":"2024-04-15T15:17:02.060829+0000","last_peered":"2024-04-15T15:17:02.060829+0000","last_clean":"2024-04-15T15:17:02.060829+0000","last_became_active":"2024-04-15T15:17:00.053980+0000","last_became_peered":"2024-04-15T15:17:00.053980+0000","last_unstale":"2024-04-15T15:17:02.060829+0000","last_undegraded":"2024-04-15T15:17:02.060829+0000","last_fullsized":"2024-04-15T15:17:02.060829+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:23.176250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378093+0000","last_change":"2024-04-15T15:17:01.062667+0000","last_active":"2024-04-15T15:17:02.378093+0000","last_peered":"2024-04-15T15:17:02.378093+0000","last_clean":"2024-04-15T15:17:02.378093+0000","last_became_active":"2024-04-15T15:17:01.062171+0000","last_became_peered":"2024-04-15T15:17:01.062171+0000","last_unstale":"2024-04-15T15:17:02.378093+0000","last_undegraded":"2024-04-15T15:17:02.378093+0000","last_fullsized":"2024-04-15T15:17:02.378093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:23.651512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.080694+0000","last_change":"2024-04-15T15:17:00.058378+0000","last_active":"2024-04-15T15:17:01.080694+0000","last_peered":"2024-04-15T15:17:01.080694+0000","last_clean":"2024-04-15T15:17:01.080694+0000","last_became_active":"2024-04-15T15:17:00.058024+0000","last_became_peered":"2024-04-15T15:17:00.058024+0000","last_unstale":"2024-04-15T15:17:01.080694+0000","last_undegraded":"2024-04-15T15:17:01.080694+0000","last_fullsized":"2024-04-15T15:17:01.080694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377353+0000","last_change":"2024-04-15T15:17:01.071377+0000","last_active":"2024-04-15T15:17:02.377353+0000","last_peered":"2024-04-15T15:17:02.377353+0000","last_clean":"2024-04-15T15:17:02.377353+0000","last_became_active":"2024-04-15T15:17:01.071087+0000","last_became_peered":"2024-04-15T15:17:01.071087+0000","last_unstale":"2024-04-15T15:17:02.377353+0000","last_undegraded":"2024-04-15T15:17:02.377353+0000","last_fullsized":"2024-04-15T15:17:02.377353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.077676+0000","last_change":"2024-04-15T15:17:00.052588+0000","last_active":"2024-04-15T15:17:01.077676+0000","last_peered":"2024-04-15T15:17:01.077676+0000","last_clean":"2024-04-15T15:17:01.077676+0000","last_became_active":"2024-04-15T15:17:00.052287+0000","last_became_peered":"2024-04-15T15:17:00.052287+0000","last_unstale":"2024-04-15T15:17:01.077676+0000","last_undegraded":"2024-04-15T15:17:01.077676+0000","last_fullsized":"2024-04-15T15:17:01.077676+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079604+0000","last_change":"2024-04-15T15:17:01.077529+0000","last_active":"2024-04-15T15:17:01.079604+0000","last_peered":"2024-04-15T15:17:01.079604+0000","last_clean":"2024-04-15T15:17:01.079604+0000","last_became_active":"2024-04-15T15:17:01.077231+0000","last_became_peered":"2024-04-15T15:17:01.077231+0000","last_unstale":"2024-04-15T15:17:01.079604+0000","last_undegraded":"2024-04-15T15:17:01.079604+0000","last_fullsized":"2024-04-15T15:17:01.079604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:58:51.072457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378089+0000","last_change":"2024-04-15T15:17:00.060924+0000","last_active":"2024-04-15T15:17:02.378089+0000","last_peered":"2024-04-15T15:17:02.378089+0000","last_clean":"2024-04-15T15:17:02.378089+0000","last_became_active":"2024-04-15T15:17:00.060610+0000","last_became_peered":"2024-04-15T15:17:00.060610+0000","last_unstale":"2024-04-15T15:17:02.378089+0000","last_undegraded":"2024-04-15T15:17:02.378089+0000","last_fullsized":"2024-04-15T15:17:02.378089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:23.176250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072370+0000","last_change":"2024-04-15T15:17:01.068809+0000","last_active":"2024-04-15T15:17:01.072370+0000","last_peered":"2024-04-15T15:17:01.072370+0000","last_clean":"2024-04-15T15:17:01.072370+0000","last_became_active":"2024-04-15T15:17:01.067740+0000","last_became_peered":"2024-04-15T15:17:01.067740+0000","last_unstale":"2024-04-15T15:17:01.072370+0000","last_undegraded":"2024-04-15T15:17:01.072370+0000","last_fullsized":"2024-04-15T15:17:01.072370+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060464+0000","last_change":"2024-04-15T15:17:00.056079+0000","last_active":"2024-04-15T15:17:02.060464+0000","last_peered":"2024-04-15T15:17:02.060464+0000","last_clean":"2024-04-15T15:17:02.060464+0000","last_became_active":"2024-04-15T15:17:00.055794+0000","last_became_peered":"2024-04-15T15:17:00.055794+0000","last_unstale":"2024-04-15T15:17:02.060464+0000","last_undegraded":"2024-04-15T15:17:02.060464+0000","last_fullsized":"2024-04-15T15:17:02.060464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:56.456359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377746+0000","last_change":"2024-04-15T15:17:01.071336+0000","last_active":"2024-04-15T15:17:02.377746+0000","last_peered":"2024-04-15T15:17:02.377746+0000","last_clean":"2024-04-15T15:17:02.377746+0000","last_became_active":"2024-04-15T15:17:01.070764+0000","last_became_peered":"2024-04-15T15:17:01.070764+0000","last_unstale":"2024-04-15T15:17:02.377746+0000","last_undegraded":"2024-04-15T15:17:02.377746+0000","last_fullsized":"2024-04-15T15:17:02.377746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:24.305106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056679+0000","last_change":"2024-04-15T15:17:00.048501+0000","last_active":"2024-04-15T15:17:01.056679+0000","last_peered":"2024-04-15T15:17:01.056679+0000","last_clean":"2024-04-15T15:17:01.056679+0000","last_became_active":"2024-04-15T15:17:00.048148+0000","last_became_peered":"2024-04-15T15:17:00.048148+0000","last_unstale":"2024-04-15T15:17:01.056679+0000","last_undegraded":"2024-04-15T15:17:01.056679+0000","last_fullsized":"2024-04-15T15:17:01.056679+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075491+0000","last_change":"2024-04-15T15:17:01.071813+0000","last_active":"2024-04-15T15:17:02.075491+0000","last_peered":"2024-04-15T15:17:02.075491+0000","last_clean":"2024-04-15T15:17:02.075491+0000","last_became_active":"2024-04-15T15:17:01.071521+0000","last_became_peered":"2024-04-15T15:17:01.071521+0000","last_unstale":"2024-04-15T15:17:02.075491+0000","last_undegraded":"2024-04-15T15:17:02.075491+0000","last_fullsized":"2024-04-15T15:17:02.075491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378154+0000","last_change":"2024-04-15T15:17:00.059059+0000","last_active":"2024-04-15T15:17:02.378154+0000","last_peered":"2024-04-15T15:17:02.378154+0000","last_clean":"2024-04-15T15:17:02.378154+0000","last_became_active":"2024-04-15T15:17:00.058527+0000","last_became_peered":"2024-04-15T15:17:00.058527+0000","last_unstale":"2024-04-15T15:17:02.378154+0000","last_undegraded":"2024-04-15T15:17:02.378154+0000","last_fullsized":"2024-04-15T15:17:02.378154+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378300+0000","last_change":"2024-04-15T15:17:01.063386+0000","last_active":"2024-04-15T15:17:02.378300+0000","last_peered":"2024-04-15T15:17:02.378300+0000","last_clean":"2024-04-15T15:17:02.378300+0000","last_became_active":"2024-04-15T15:17:01.062362+0000","last_became_peered":"2024-04-15T15:17:01.062362+0000","last_unstale":"2024-04-15T15:17:02.378300+0000","last_undegraded":"2024-04-15T15:17:02.378300+0000","last_fullsized":"2024-04-15T15:17:02.378300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:46.256635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.056094+0000","last_change":"2024-04-15T15:17:00.058594+0000","last_active":"2024-04-15T15:17:01.056094+0000","last_peered":"2024-04-15T15:17:01.056094+0000","last_clean":"2024-04-15T15:17:01.056094+0000","last_became_active":"2024-04-15T15:17:00.058359+0000","last_became_peered":"2024-04-15T15:17:00.058359+0000","last_unstale":"2024-04-15T15:17:01.056094+0000","last_undegraded":"2024-04-15T15:17:01.056094+0000","last_fullsized":"2024-04-15T15:17:01.056094+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377781+0000","last_change":"2024-04-15T15:17:01.065464+0000","last_active":"2024-04-15T15:17:02.377781+0000","last_peered":"2024-04-15T15:17:02.377781+0000","last_clean":"2024-04-15T15:17:02.377781+0000","last_became_active":"2024-04-15T15:17:01.065142+0000","last_became_peered":"2024-04-15T15:17:01.065142+0000","last_unstale":"2024-04-15T15:17:02.377781+0000","last_undegraded":"2024-04-15T15:17:02.377781+0000","last_fullsized":"2024-04-15T15:17:02.377781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078903+0000","last_change":"2024-04-15T15:17:00.055108+0000","last_active":"2024-04-15T15:17:01.078903+0000","last_peered":"2024-04-15T15:17:01.078903+0000","last_clean":"2024-04-15T15:17:01.078903+0000","last_became_active":"2024-04-15T15:17:00.054818+0000","last_became_peered":"2024-04-15T15:17:00.054818+0000","last_unstale":"2024-04-15T15:17:01.078903+0000","last_undegraded":"2024-04-15T15:17:01.078903+0000","last_fullsized":"2024-04-15T15:17:01.078903+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072041+0000","last_change":"2024-04-15T15:17:01.066511+0000","last_active":"2024-04-15T15:17:01.072041+0000","last_peered":"2024-04-15T15:17:01.072041+0000","last_clean":"2024-04-15T15:17:01.072041+0000","last_became_active":"2024-04-15T15:17:01.066201+0000","last_became_peered":"2024-04-15T15:17:01.066201+0000","last_unstale":"2024-04-15T15:17:01.072041+0000","last_undegraded":"2024-04-15T15:17:01.072041+0000","last_fullsized":"2024-04-15T15:17:01.072041+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377819+0000","last_change":"2024-04-15T15:17:00.061109+0000","last_active":"2024-04-15T15:17:02.377819+0000","last_peered":"2024-04-15T15:17:02.377819+0000","last_clean":"2024-04-15T15:17:02.377819+0000","last_became_active":"2024-04-15T15:17:00.060823+0000","last_became_peered":"2024-04-15T15:17:00.060823+0000","last_unstale":"2024-04-15T15:17:02.377819+0000","last_undegraded":"2024-04-15T15:17:02.377819+0000","last_fullsized":"2024-04-15T15:17:02.377819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:36.769094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377423+0000","last_change":"2024-04-15T15:17:01.067591+0000","last_active":"2024-04-15T15:17:02.377423+0000","last_peered":"2024-04-15T15:17:02.377423+0000","last_clean":"2024-04-15T15:17:02.377423+0000","last_became_active":"2024-04-15T15:17:01.065638+0000","last_became_peered":"2024-04-15T15:17:01.065638+0000","last_unstale":"2024-04-15T15:17:02.377423+0000","last_undegraded":"2024-04-15T15:17:02.377423+0000","last_fullsized":"2024-04-15T15:17:02.377423+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377974+0000","last_change":"2024-04-15T15:17:00.057661+0000","last_active":"2024-04-15T15:17:02.377974+0000","last_peered":"2024-04-15T15:17:02.377974+0000","last_clean":"2024-04-15T15:17:02.377974+0000","last_became_active":"2024-04-15T15:17:00.057376+0000","last_became_peered":"2024-04-15T15:17:00.057376+0000","last_unstale":"2024-04-15T15:17:02.377974+0000","last_undegraded":"2024-04-15T15:17:02.377974+0000","last_fullsized":"2024-04-15T15:17:02.377974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377664+0000","last_change":"2024-04-15T15:17:01.061592+0000","last_active":"2024-04-15T15:17:02.377664+0000","last_peered":"2024-04-15T15:17:02.377664+0000","last_clean":"2024-04-15T15:17:02.377664+0000","last_became_active":"2024-04-15T15:17:01.061250+0000","last_became_peered":"2024-04-15T15:17:01.061250+0000","last_unstale":"2024-04-15T15:17:02.377664+0000","last_undegraded":"2024-04-15T15:17:02.377664+0000","last_fullsized":"2024-04-15T15:17:02.377664+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:46.256635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060563+0000","last_change":"2024-04-15T15:17:00.055785+0000","last_active":"2024-04-15T15:17:02.060563+0000","last_peered":"2024-04-15T15:17:02.060563+0000","last_clean":"2024-04-15T15:17:02.060563+0000","last_became_active":"2024-04-15T15:17:00.055509+0000","last_became_peered":"2024-04-15T15:17:00.055509+0000","last_unstale":"2024-04-15T15:17:02.060563+0000","last_undegraded":"2024-04-15T15:17:02.060563+0000","last_fullsized":"2024-04-15T15:17:02.060563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377999+0000","last_change":"2024-04-15T15:17:01.056835+0000","last_active":"2024-04-15T15:17:02.377999+0000","last_peered":"2024-04-15T15:17:02.377999+0000","last_clean":"2024-04-15T15:17:02.377999+0000","last_became_active":"2024-04-15T15:17:01.056566+0000","last_became_peered":"2024-04-15T15:17:01.056566+0000","last_unstale":"2024-04-15T15:17:02.377999+0000","last_undegraded":"2024-04-15T15:17:02.377999+0000","last_fullsized":"2024-04-15T15:17:02.377999+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048522+0000","last_change":"2024-04-15T15:17:00.055842+0000","last_active":"2024-04-15T15:17:01.048522+0000","last_peered":"2024-04-15T15:17:01.048522+0000","last_clean":"2024-04-15T15:17:01.048522+0000","last_became_active":"2024-04-15T15:17:00.054053+0000","last_became_peered":"2024-04-15T15:17:00.054053+0000","last_unstale":"2024-04-15T15:17:01.048522+0000","last_undegraded":"2024-04-15T15:17:01.048522+0000","last_fullsized":"2024-04-15T15:17:01.048522+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377880+0000","last_change":"2024-04-15T15:17:01.060443+0000","last_active":"2024-04-15T15:17:02.377880+0000","last_peered":"2024-04-15T15:17:02.377880+0000","last_clean":"2024-04-15T15:17:02.377880+0000","last_became_active":"2024-04-15T15:17:01.060102+0000","last_became_peered":"2024-04-15T15:17:01.060102+0000","last_unstale":"2024-04-15T15:17:02.377880+0000","last_undegraded":"2024-04-15T15:17:02.377880+0000","last_fullsized":"2024-04-15T15:17:02.377880+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:08.601149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377886+0000","last_change":"2024-04-15T15:17:00.061010+0000","last_active":"2024-04-15T15:17:02.377886+0000","last_peered":"2024-04-15T15:17:02.377886+0000","last_clean":"2024-04-15T15:17:02.377886+0000","last_became_active":"2024-04-15T15:17:00.060723+0000","last_became_peered":"2024-04-15T15:17:00.060723+0000","last_unstale":"2024-04-15T15:17:02.377886+0000","last_undegraded":"2024-04-15T15:17:02.377886+0000","last_fullsized":"2024-04-15T15:17:02.377886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:22.029429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377635+0000","last_change":"2024-04-15T15:17:01.060543+0000","last_active":"2024-04-15T15:17:02.377635+0000","last_peered":"2024-04-15T15:17:02.377635+0000","last_clean":"2024-04-15T15:17:02.377635+0000","last_became_active":"2024-04-15T15:17:01.060255+0000","last_became_peered":"2024-04-15T15:17:01.060255+0000","last_unstale":"2024-04-15T15:17:02.377635+0000","last_undegraded":"2024-04-15T15:17:02.377635+0000","last_fullsized":"2024-04-15T15:17:02.377635+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:47.544340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078087+0000","last_change":"2024-04-15T15:17:00.053088+0000","last_active":"2024-04-15T15:17:01.078087+0000","last_peered":"2024-04-15T15:17:01.078087+0000","last_clean":"2024-04-15T15:17:01.078087+0000","last_became_active":"2024-04-15T15:17:00.052762+0000","last_became_peered":"2024-04-15T15:17:00.052762+0000","last_unstale":"2024-04-15T15:17:01.078087+0000","last_undegraded":"2024-04-15T15:17:01.078087+0000","last_fullsized":"2024-04-15T15:17:01.078087+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.075375+0000","last_change":"2024-04-15T15:17:01.067435+0000","last_active":"2024-04-15T15:17:01.075375+0000","last_peered":"2024-04-15T15:17:01.075375+0000","last_clean":"2024-04-15T15:17:01.075375+0000","last_became_active":"2024-04-15T15:17:01.066970+0000","last_became_peered":"2024-04-15T15:17:01.066970+0000","last_unstale":"2024-04-15T15:17:01.075375+0000","last_undegraded":"2024-04-15T15:17:01.075375+0000","last_fullsized":"2024-04-15T15:17:01.075375+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.079013+0000","last_change":"2024-04-15T15:17:00.049730+0000","last_active":"2024-04-15T15:17:01.079013+0000","last_peered":"2024-04-15T15:17:01.079013+0000","last_clean":"2024-04-15T15:17:01.079013+0000","last_became_active":"2024-04-15T15:17:00.049472+0000","last_became_peered":"2024-04-15T15:17:00.049472+0000","last_unstale":"2024-04-15T15:17:01.079013+0000","last_undegraded":"2024-04-15T15:17:01.079013+0000","last_fullsized":"2024-04-15T15:17:01.079013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072086+0000","last_change":"2024-04-15T15:17:01.067474+0000","last_active":"2024-04-15T15:17:01.072086+0000","last_peered":"2024-04-15T15:17:01.072086+0000","last_clean":"2024-04-15T15:17:01.072086+0000","last_became_active":"2024-04-15T15:17:01.066802+0000","last_became_peered":"2024-04-15T15:17:01.066802+0000","last_unstale":"2024-04-15T15:17:01.072086+0000","last_undegraded":"2024-04-15T15:17:01.072086+0000","last_fullsized":"2024-04-15T15:17:01.072086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075391+0000","last_change":"2024-04-15T15:17:00.054252+0000","last_active":"2024-04-15T15:17:02.075391+0000","last_peered":"2024-04-15T15:17:02.075391+0000","last_clean":"2024-04-15T15:17:02.075391+0000","last_became_active":"2024-04-15T15:17:00.053917+0000","last_became_peered":"2024-04-15T15:17:00.053917+0000","last_unstale":"2024-04-15T15:17:02.075391+0000","last_undegraded":"2024-04-15T15:17:02.075391+0000","last_fullsized":"2024-04-15T15:17:02.075391+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:24:18.668440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075557+0000","last_change":"2024-04-15T15:17:01.058076+0000","last_active":"2024-04-15T15:17:02.075557+0000","last_peered":"2024-04-15T15:17:02.075557+0000","last_clean":"2024-04-15T15:17:02.075557+0000","last_became_active":"2024-04-15T15:17:01.057615+0000","last_became_peered":"2024-04-15T15:17:01.057615+0000","last_unstale":"2024-04-15T15:17:02.075557+0000","last_undegraded":"2024-04-15T15:17:02.075557+0000","last_fullsized":"2024-04-15T15:17:02.075557+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060631+0000","last_change":"2024-04-15T15:17:00.053573+0000","last_active":"2024-04-15T15:17:02.060631+0000","last_peered":"2024-04-15T15:17:02.060631+0000","last_clean":"2024-04-15T15:17:02.060631+0000","last_became_active":"2024-04-15T15:17:00.053296+0000","last_became_peered":"2024-04-15T15:17:00.053296+0000","last_unstale":"2024-04-15T15:17:02.060631+0000","last_undegraded":"2024-04-15T15:17:02.060631+0000","last_fullsized":"2024-04-15T15:17:02.060631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378107+0000","last_change":"2024-04-15T15:17:01.061605+0000","last_active":"2024-04-15T15:17:02.378107+0000","last_peered":"2024-04-15T15:17:02.378107+0000","last_clean":"2024-04-15T15:17:02.378107+0000","last_became_active":"2024-04-15T15:17:01.061267+0000","last_became_peered":"2024-04-15T15:17:01.061267+0000","last_unstale":"2024-04-15T15:17:02.378107+0000","last_undegraded":"2024-04-15T15:17:02.378107+0000","last_fullsized":"2024-04-15T15:17:02.378107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:58:51.072457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075228+0000","last_change":"2024-04-15T15:17:00.052685+0000","last_active":"2024-04-15T15:17:02.075228+0000","last_peered":"2024-04-15T15:17:02.075228+0000","last_clean":"2024-04-15T15:17:02.075228+0000","last_became_active":"2024-04-15T15:17:00.052398+0000","last_became_peered":"2024-04-15T15:17:00.052398+0000","last_unstale":"2024-04-15T15:17:02.075228+0000","last_undegraded":"2024-04-15T15:17:02.075228+0000","last_fullsized":"2024-04-15T15:17:02.075228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:30:51.941942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377775+0000","last_change":"2024-04-15T15:17:01.064764+0000","last_active":"2024-04-15T15:17:02.377775+0000","last_peered":"2024-04-15T15:17:02.377775+0000","last_clean":"2024-04-15T15:17:02.377775+0000","last_became_active":"2024-04-15T15:17:01.064496+0000","last_became_peered":"2024-04-15T15:17:01.064496+0000","last_unstale":"2024-04-15T15:17:02.377775+0000","last_undegraded":"2024-04-15T15:17:02.377775+0000","last_fullsized":"2024-04-15T15:17:02.377775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075244+0000","last_change":"2024-04-15T15:17:00.044538+0000","last_active":"2024-04-15T15:17:02.075244+0000","last_peered":"2024-04-15T15:17:02.075244+0000","last_clean":"2024-04-15T15:17:02.075244+0000","last_became_active":"2024-04-15T15:17:00.044237+0000","last_became_peered":"2024-04-15T15:17:00.044237+0000","last_unstale":"2024-04-15T15:17:02.075244+0000","last_undegraded":"2024-04-15T15:17:02.075244+0000","last_fullsized":"2024-04-15T15:17:02.075244+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:16:18.037173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.073110+0000","last_change":"2024-04-15T15:17:01.066478+0000","last_active":"2024-04-15T15:17:01.073110+0000","last_peered":"2024-04-15T15:17:01.073110+0000","last_clean":"2024-04-15T15:17:01.073110+0000","last_became_active":"2024-04-15T15:17:01.066229+0000","last_became_peered":"2024-04-15T15:17:01.066229+0000","last_unstale":"2024-04-15T15:17:01.073110+0000","last_undegraded":"2024-04-15T15:17:01.073110+0000","last_fullsized":"2024-04-15T15:17:01.073110+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378194+0000","last_change":"2024-04-15T15:17:00.064123+0000","last_active":"2024-04-15T15:17:02.378194+0000","last_peered":"2024-04-15T15:17:02.378194+0000","last_clean":"2024-04-15T15:17:02.378194+0000","last_became_active":"2024-04-15T15:17:00.063535+0000","last_became_peered":"2024-04-15T15:17:00.063535+0000","last_unstale":"2024-04-15T15:17:02.378194+0000","last_undegraded":"2024-04-15T15:17:02.378194+0000","last_fullsized":"2024-04-15T15:17:02.378194+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378009+0000","last_change":"2024-04-15T15:17:01.071081+0000","last_active":"2024-04-15T15:17:02.378009+0000","last_peered":"2024-04-15T15:17:02.378009+0000","last_clean":"2024-04-15T15:17:02.378009+0000","last_became_active":"2024-04-15T15:17:01.070378+0000","last_became_peered":"2024-04-15T15:17:01.070378+0000","last_unstale":"2024-04-15T15:17:02.378009+0000","last_undegraded":"2024-04-15T15:17:02.378009+0000","last_fullsized":"2024-04-15T15:17:02.378009+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:19:19.245967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075171+0000","last_change":"2024-04-15T15:17:00.051113+0000","last_active":"2024-04-15T15:17:02.075171+0000","last_peered":"2024-04-15T15:17:02.075171+0000","last_clean":"2024-04-15T15:17:02.075171+0000","last_became_active":"2024-04-15T15:17:00.050824+0000","last_became_peered":"2024-04-15T15:17:00.050824+0000","last_unstale":"2024-04-15T15:17:02.075171+0000","last_undegraded":"2024-04-15T15:17:02.075171+0000","last_fullsized":"2024-04-15T15:17:02.075171+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:35:25.744871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060631+0000","last_change":"2024-04-15T15:17:01.067058+0000","last_active":"2024-04-15T15:17:02.060631+0000","last_peered":"2024-04-15T15:17:02.060631+0000","last_clean":"2024-04-15T15:17:02.060631+0000","last_became_active":"2024-04-15T15:17:01.066787+0000","last_became_peered":"2024-04-15T15:17:01.066787+0000","last_unstale":"2024-04-15T15:17:02.060631+0000","last_undegraded":"2024-04-15T15:17:02.060631+0000","last_fullsized":"2024-04-15T15:17:02.060631+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.078424+0000","last_change":"2024-04-15T15:17:00.058333+0000","last_active":"2024-04-15T15:17:01.078424+0000","last_peered":"2024-04-15T15:17:01.078424+0000","last_clean":"2024-04-15T15:17:01.078424+0000","last_became_active":"2024-04-15T15:17:00.057892+0000","last_became_peered":"2024-04-15T15:17:00.057892+0000","last_unstale":"2024-04-15T15:17:01.078424+0000","last_undegraded":"2024-04-15T15:17:01.078424+0000","last_fullsized":"2024-04-15T15:17:01.078424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378605+0000","last_change":"2024-04-15T15:17:01.062606+0000","last_active":"2024-04-15T15:17:02.378605+0000","last_peered":"2024-04-15T15:17:02.378605+0000","last_clean":"2024-04-15T15:17:02.378605+0000","last_became_active":"2024-04-15T15:17:01.062062+0000","last_became_peered":"2024-04-15T15:17:01.062062+0000","last_unstale":"2024-04-15T15:17:02.378605+0000","last_undegraded":"2024-04-15T15:17:02.378605+0000","last_fullsized":"2024-04-15T15:17:02.378605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:08.601149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075486+0000","last_change":"2024-04-15T15:17:00.049576+0000","last_active":"2024-04-15T15:17:02.075486+0000","last_peered":"2024-04-15T15:17:02.075486+0000","last_clean":"2024-04-15T15:17:02.075486+0000","last_became_active":"2024-04-15T15:17:00.049280+0000","last_became_peered":"2024-04-15T15:17:00.049280+0000","last_unstale":"2024-04-15T15:17:02.075486+0000","last_undegraded":"2024-04-15T15:17:02.075486+0000","last_fullsized":"2024-04-15T15:17:02.075486+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:15:28.100508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378101+0000","last_change":"2024-04-15T15:17:01.069695+0000","last_active":"2024-04-15T15:17:02.378101+0000","last_peered":"2024-04-15T15:17:02.378101+0000","last_clean":"2024-04-15T15:17:02.378101+0000","last_became_active":"2024-04-15T15:17:01.069323+0000","last_became_peered":"2024-04-15T15:17:01.069323+0000","last_unstale":"2024-04-15T15:17:02.378101+0000","last_undegraded":"2024-04-15T15:17:02.378101+0000","last_fullsized":"2024-04-15T15:17:02.378101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:42:24.767991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377573+0000","last_change":"2024-04-15T15:17:00.061061+0000","last_active":"2024-04-15T15:17:02.377573+0000","last_peered":"2024-04-15T15:17:02.377573+0000","last_clean":"2024-04-15T15:17:02.377573+0000","last_became_active":"2024-04-15T15:17:00.060731+0000","last_became_peered":"2024-04-15T15:17:00.060731+0000","last_unstale":"2024-04-15T15:17:02.377573+0000","last_undegraded":"2024-04-15T15:17:02.377573+0000","last_fullsized":"2024-04-15T15:17:02.377573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:56.456359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377520+0000","last_change":"2024-04-15T15:17:01.065896+0000","last_active":"2024-04-15T15:17:02.377520+0000","last_peered":"2024-04-15T15:17:02.377520+0000","last_clean":"2024-04-15T15:17:02.377520+0000","last_became_active":"2024-04-15T15:17:01.065259+0000","last_became_peered":"2024-04-15T15:17:01.065259+0000","last_unstale":"2024-04-15T15:17:02.377520+0000","last_undegraded":"2024-04-15T15:17:02.377520+0000","last_fullsized":"2024-04-15T15:17:02.377520+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377628+0000","last_change":"2024-04-15T15:17:00.063764+0000","last_active":"2024-04-15T15:17:02.377628+0000","last_peered":"2024-04-15T15:17:02.377628+0000","last_clean":"2024-04-15T15:17:02.377628+0000","last_became_active":"2024-04-15T15:17:00.063528+0000","last_became_peered":"2024-04-15T15:17:00.063528+0000","last_unstale":"2024-04-15T15:17:02.377628+0000","last_undegraded":"2024-04-15T15:17:02.377628+0000","last_fullsized":"2024-04-15T15:17:02.377628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060612+0000","last_change":"2024-04-15T15:17:01.070588+0000","last_active":"2024-04-15T15:17:02.060612+0000","last_peered":"2024-04-15T15:17:02.060612+0000","last_clean":"2024-04-15T15:17:02.060612+0000","last_became_active":"2024-04-15T15:17:01.070263+0000","last_became_peered":"2024-04-15T15:17:01.070263+0000","last_unstale":"2024-04-15T15:17:02.060612+0000","last_undegraded":"2024-04-15T15:17:02.060612+0000","last_fullsized":"2024-04-15T15:17:02.060612+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.048642+0000","last_change":"2024-04-15T15:17:00.056627+0000","last_active":"2024-04-15T15:17:01.048642+0000","last_peered":"2024-04-15T15:17:01.048642+0000","last_clean":"2024-04-15T15:17:01.048642+0000","last_became_active":"2024-04-15T15:17:00.056317+0000","last_became_peered":"2024-04-15T15:17:00.056317+0000","last_unstale":"2024-04-15T15:17:01.048642+0000","last_undegraded":"2024-04-15T15:17:01.048642+0000","last_fullsized":"2024-04-15T15:17:01.048642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074263+0000","last_change":"2024-04-15T15:17:01.071745+0000","last_active":"2024-04-15T15:17:01.074263+0000","last_peered":"2024-04-15T15:17:01.074263+0000","last_clean":"2024-04-15T15:17:01.074263+0000","last_became_active":"2024-04-15T15:17:01.071466+0000","last_became_peered":"2024-04-15T15:17:01.071466+0000","last_unstale":"2024-04-15T15:17:01.074263+0000","last_undegraded":"2024-04-15T15:17:01.074263+0000","last_fullsized":"2024-04-15T15:17:01.074263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:04.108334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.054345+0000","last_change":"2024-04-15T15:17:00.057863+0000","last_active":"2024-04-15T15:17:01.054345+0000","last_peered":"2024-04-15T15:17:01.054345+0000","last_clean":"2024-04-15T15:17:01.054345+0000","last_became_active":"2024-04-15T15:17:00.057596+0000","last_became_peered":"2024-04-15T15:17:00.057596+0000","last_unstale":"2024-04-15T15:17:01.054345+0000","last_undegraded":"2024-04-15T15:17:01.054345+0000","last_fullsized":"2024-04-15T15:17:01.054345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378193+0000","last_change":"2024-04-15T15:17:01.060073+0000","last_active":"2024-04-15T15:17:02.378193+0000","last_peered":"2024-04-15T15:17:02.378193+0000","last_clean":"2024-04-15T15:17:02.378193+0000","last_became_active":"2024-04-15T15:17:01.059802+0000","last_became_peered":"2024-04-15T15:17:01.059802+0000","last_unstale":"2024-04-15T15:17:02.378193+0000","last_undegraded":"2024-04-15T15:17:02.378193+0000","last_fullsized":"2024-04-15T15:17:02.378193+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:47.544340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377563+0000","last_change":"2024-04-15T15:17:00.058811+0000","last_active":"2024-04-15T15:17:02.377563+0000","last_peered":"2024-04-15T15:17:02.377563+0000","last_clean":"2024-04-15T15:17:02.377563+0000","last_became_active":"2024-04-15T15:17:00.058530+0000","last_became_peered":"2024-04-15T15:17:00.058530+0000","last_unstale":"2024-04-15T15:17:02.377563+0000","last_undegraded":"2024-04-15T15:17:02.377563+0000","last_fullsized":"2024-04-15T15:17:02.377563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.074626+0000","last_change":"2024-04-15T15:17:01.070460+0000","last_active":"2024-04-15T15:17:01.074626+0000","last_peered":"2024-04-15T15:17:01.074626+0000","last_clean":"2024-04-15T15:17:01.074626+0000","last_became_active":"2024-04-15T15:17:01.070110+0000","last_became_peered":"2024-04-15T15:17:01.070110+0000","last_unstale":"2024-04-15T15:17:01.074626+0000","last_undegraded":"2024-04-15T15:17:01.074626+0000","last_fullsized":"2024-04-15T15:17:01.074626+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378282+0000","last_change":"2024-04-15T15:17:00.061284+0000","last_active":"2024-04-15T15:17:02.378282+0000","last_peered":"2024-04-15T15:17:02.378282+0000","last_clean":"2024-04-15T15:17:02.378282+0000","last_became_active":"2024-04-15T15:17:00.060520+0000","last_became_peered":"2024-04-15T15:17:00.060520+0000","last_unstale":"2024-04-15T15:17:02.378282+0000","last_undegraded":"2024-04-15T15:17:02.378282+0000","last_fullsized":"2024-04-15T15:17:02.378282+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:03.103510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":202,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378054+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:02.378054+0000","last_peered":"2024-04-15T15:17:02.378054+0000","last_clean":"2024-04-15T15:17:02.378054+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:02.378054+0000","last_undegraded":"2024-04-15T15:17:02.378054+0000","last_fullsized":"2024-04-15T15:17:02.378054+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":438272,"data_stored":187741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":187763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1056768,"data_stored":785901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":462848,"data_stored":195497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":450560,"data_stored":193921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":450560,"data_stored":193941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1044480,"data_stored":784253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:09.560 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.675+0000 7f13ccad9640 1 Processor -- start 2024-04-15T15:17:09.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.675+0000 7f13ccad9640 1 -- start start 2024-04-15T15:17:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13ccad9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13ccad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13c815fb30 0x7f13c80d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13ccad9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c8070720 con 0x7f13c80701f0 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13ccad9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f13c809ba90 con 0x7f13c815fb30 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13ccad9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f13c80d08c0 con 0x7f13c809a9d0 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13c6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f13c815fb30 0x7f13c80d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13bffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f13c80701f0 0x7f13c809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48818/0 (socket says 172.21.15.46:48818) 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13bffff640 1 -- 172.21.15.46:0/2422922813 learned_addr learned my addr 172.21.15.46:0/2422922813 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13c659c640 1 --2- 172.21.15.46:0/2422922813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2422922813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 msgr2=0x7f13c8160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13c6d9d640 1 --2- 172.21.15.46:0/2422922813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c8160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2422922813 >> v1:172.21.15.46:6789/0 conn(0x7f13c80701f0 legacy=0x7f13c809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.676+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2422922813 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c8070a90 con 0x7f13c815fb30 2024-04-15T15:17:09.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.677+0000 7f13c6d9d640 1 --2- 172.21.15.46:0/2422922813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80d0d50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009dc0 tx=0x7f13c002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1858307d2a4e9767 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.677+0000 7f13c5d9b640 1 -- 172.21.15.46:0/2422922813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0039070 con 0x7f13c815fb30 2024-04-15T15:17:09.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.677+0000 7f13c5d9b640 1 -- 172.21.15.46:0/2422922813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c002cc40 con 0x7f13c815fb30 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.677+0000 7f13c5d9b640 1 -- 172.21.15.46:0/2422922813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0034440 con 0x7f13c815fb30 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 -- 172.21.15.46:0/2422922813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 msgr2=0x7f13c80d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2422922813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80d0d50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009dc0 tx=0x7f13c002be90 comp rx=0 tx=0).stop 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 -- 172.21.15.46:0/2422922813 shutdown_connections 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2422922813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80d0d50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2422922813 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c8160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 -- 172.21.15.46:0/2422922813 >> 172.21.15.46:0/2422922813 conn(0x7f13c80d2460 msgr2=0x7f13c80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 -- 172.21.15.46:0/2422922813 shutdown_connections 2024-04-15T15:17:09.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.678+0000 7f13ccad9640 1 -- 172.21.15.46:0/2422922813 wait complete. 2024-04-15T15:17:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.679+0000 7f13ccad9640 1 Processor -- start 2024-04-15T15:17:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.679+0000 7f13ccad9640 1 -- start start 2024-04-15T15:17:09.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13ccad9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13c80701f0 0x7f13c8154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13ccad9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c80cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13ccad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13ccad9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13c815ffc0 con 0x7f13c815fb30 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13ccad9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f13c8071050 con 0x7f13c80701f0 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13ccad9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f13c80b1c60 con 0x7f13c809a9d0 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13c6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59608/0 (socket says 172.21.15.46:59608) 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2608136121 learned_addr learned my addr 172.21.15.46:0/2608136121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13bffff640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c80cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.680+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 msgr2=0x7f13c80cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c6d9d640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c80cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13c80701f0 msgr2=0x7f13c8154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c659c640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13c80701f0 0x7f13c8154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c6d9d640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13c80701f0 0x7f13c8154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c6d9d640 1 -- 172.21.15.46:0/2608136121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c80b02f0 con 0x7f13c815fb30 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13bffff640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c80cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c659c640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13c80701f0 0x7f13c8154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13c6d9d640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80cdfe0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009ee0 tx=0x7f13c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0039050 con 0x7f13c815fb30 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13c0045070 con 0x7f13c815fb30 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c80b05d0 con 0x7f13c815fb30 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c0040690 con 0x7f13c815fb30 2024-04-15T15:17:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.681+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c80b0b90 con 0x7f13c815fb30 2024-04-15T15:17:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.682+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c80689e0 con 0x7f13c815fb30 2024-04-15T15:17:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.683+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13c0040830 con 0x7f13c815fb30 2024-04-15T15:17:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.683+0000 7f13beffd640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13a806c8f0 0x7f13a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.683+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f13c00ae7c0 con 0x7f13c815fb30 2024-04-15T15:17:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.683+0000 7f13c659c640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13a806c8f0 0x7f13a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.684+0000 7f13c659c640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13a806c8f0 0x7f13a806edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f13b8006fd0 tx=0x7f13b8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.688+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13c007ca60 con 0x7f13c815fb30 2024-04-15T15:17:09.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.868+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13c81634a0 con 0x7f13a806c8f0 2024-04-15T15:17:09.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.875+0000 7f13beffd640 1 -- 172.21.15.46:0/2608136121 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251935 (secure 0 0 0) 0x7f13c81634a0 con 0x7f13a806c8f0 2024-04-15T15:17:09.879 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:09.880 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13a806c8f0 msgr2=0x7f13a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13a806c8f0 0x7f13a806edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f13b8006fd0 tx=0x7f13b8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 msgr2=0x7f13c80cdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80cdfe0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009ee0 tx=0x7f13c0004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 shutdown_connections 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f13a806c8f0 0x7f13a806edb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13c815fb30 0x7f13c80cdfe0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f13c809a9d0 0x7f13c80cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 --2- 172.21.15.46:0/2608136121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13c80701f0 0x7f13c8154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 >> 172.21.15.46:0/2608136121 conn(0x7f13c80d2460 msgr2=0x7f13c815e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 shutdown_connections 2024-04-15T15:17:09.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:09.878+0000 7f13ccad9640 1 -- 172.21.15.46:0/2608136121 wait complete. 2024-04-15T15:17:09.891 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-15T15:17:08.374125+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237560,"kb_used_data":4712,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737160,"statfs":{"total":773094113280,"available":772850851840,"internally_reserved":0,"allocated":4825088,"data_stored":2736364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001837"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378116+0000","last_change":"2024-04-15T15:17:01.062449+0000","last_active":"2024-04-15T15:17:02.378116+0000","last_peered":"2024-04-15T15:17:02.378116+0000","last_clean":"2024-04-15T15:17:02.378116+0000","last_became_active":"2024-04-15T15:17:01.060871+0000","last_became_peered":"2024-04-15T15:17:01.060871+0000","last_unstale":"2024-04-15T15:17:02.378116+0000","last_undegraded":"2024-04-15T15:17:02.378116+0000","last_fullsized":"2024-04-15T15:17:02.378116+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:44.232483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378008+0000","last_change":"2024-04-15T15:17:01.058267+0000","last_active":"2024-04-15T15:17:02.378008+0000","last_peered":"2024-04-15T15:17:02.378008+0000","last_clean":"2024-04-15T15:17:02.378008+0000","last_became_active":"2024-04-15T15:17:01.057296+0000","last_became_peered":"2024-04-15T15:17:01.057296+0000","last_unstale":"2024-04-15T15:17:02.378008+0000","last_undegraded":"2024-04-15T15:17:02.378008+0000","last_fullsized":"2024-04-15T15:17:02.378008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:07:59.866764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377486+0000","last_change":"2024-04-15T15:17:01.067156+0000","last_active":"2024-04-15T15:17:02.377486+0000","last_peered":"2024-04-15T15:17:02.377486+0000","last_clean":"2024-04-15T15:17:02.377486+0000","last_became_active":"2024-04-15T15:17:01.066771+0000","last_became_peered":"2024-04-15T15:17:01.066771+0000","last_unstale":"2024-04-15T15:17:02.377486+0000","last_undegraded":"2024-04-15T15:17:02.377486+0000","last_fullsized":"2024-04-15T15:17:02.377486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075545+0000","last_change":"2024-04-15T15:17:01.071566+0000","last_active":"2024-04-15T15:17:02.075545+0000","last_peered":"2024-04-15T15:17:02.075545+0000","last_clean":"2024-04-15T15:17:02.075545+0000","last_became_active":"2024-04-15T15:17:01.071282+0000","last_became_peered":"2024-04-15T15:17:01.071282+0000","last_unstale":"2024-04-15T15:17:02.075545+0000","last_undegraded":"2024-04-15T15:17:02.075545+0000","last_fullsized":"2024-04-15T15:17:02.075545+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:18.261084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075285+0000","last_change":"2024-04-15T15:17:01.071320+0000","last_active":"2024-04-15T15:17:02.075285+0000","last_peered":"2024-04-15T15:17:02.075285+0000","last_clean":"2024-04-15T15:17:02.075285+0000","last_became_active":"2024-04-15T15:17:01.071021+0000","last_became_peered":"2024-04-15T15:17:01.071021+0000","last_unstale":"2024-04-15T15:17:02.075285+0000","last_undegraded":"2024-04-15T15:17:02.075285+0000","last_fullsized":"2024-04-15T15:17:02.075285+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:59.380296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060471+0000","last_change":"2024-04-15T15:17:01.064800+0000","last_active":"2024-04-15T15:17:02.060471+0000","last_peered":"2024-04-15T15:17:02.060471+0000","last_clean":"2024-04-15T15:17:02.060471+0000","last_became_active":"2024-04-15T15:17:01.064425+0000","last_became_peered":"2024-04-15T15:17:01.064425+0000","last_unstale":"2024-04-15T15:17:02.060471+0000","last_undegraded":"2024-04-15T15:17:02.060471+0000","last_fullsized":"2024-04-15T15:17:02.060471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080085+0000","last_change":"2024-04-15T15:17:01.069002+0000","last_active":"2024-04-15T15:17:02.080085+0000","last_peered":"2024-04-15T15:17:02.080085+0000","last_clean":"2024-04-15T15:17:02.080085+0000","last_became_active":"2024-04-15T15:17:01.068755+0000","last_became_peered":"2024-04-15T15:17:01.068755+0000","last_unstale":"2024-04-15T15:17:02.080085+0000","last_undegraded":"2024-04-15T15:17:02.080085+0000","last_fullsized":"2024-04-15T15:17:02.080085+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:47.544340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377473+0000","last_change":"2024-04-15T15:17:01.071878+0000","last_active":"2024-04-15T15:17:02.377473+0000","last_peered":"2024-04-15T15:17:02.377473+0000","last_clean":"2024-04-15T15:17:02.377473+0000","last_became_active":"2024-04-15T15:17:01.070505+0000","last_became_peered":"2024-04-15T15:17:01.070505+0000","last_unstale":"2024-04-15T15:17:02.377473+0000","last_undegraded":"2024-04-15T15:17:02.377473+0000","last_fullsized":"2024-04-15T15:17:02.377473+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:25.547677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378040+0000","last_change":"2024-04-15T15:17:01.066415+0000","last_active":"2024-04-15T15:17:02.378040+0000","last_peered":"2024-04-15T15:17:02.378040+0000","last_clean":"2024-04-15T15:17:02.378040+0000","last_became_active":"2024-04-15T15:17:01.066137+0000","last_became_peered":"2024-04-15T15:17:01.066137+0000","last_unstale":"2024-04-15T15:17:02.378040+0000","last_undegraded":"2024-04-15T15:17:02.378040+0000","last_fullsized":"2024-04-15T15:17:02.378040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080448+0000","last_change":"2024-04-15T15:17:01.068258+0000","last_active":"2024-04-15T15:17:02.080448+0000","last_peered":"2024-04-15T15:17:02.080448+0000","last_clean":"2024-04-15T15:17:02.080448+0000","last_became_active":"2024-04-15T15:17:01.067888+0000","last_became_peered":"2024-04-15T15:17:01.067888+0000","last_unstale":"2024-04-15T15:17:02.080448+0000","last_undegraded":"2024-04-15T15:17:02.080448+0000","last_fullsized":"2024-04-15T15:17:02.080448+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:07:59.866764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080467+0000","last_change":"2024-04-15T15:17:01.070976+0000","last_active":"2024-04-15T15:17:02.080467+0000","last_peered":"2024-04-15T15:17:02.080467+0000","last_clean":"2024-04-15T15:17:02.080467+0000","last_became_active":"2024-04-15T15:17:01.070678+0000","last_became_peered":"2024-04-15T15:17:01.070678+0000","last_unstale":"2024-04-15T15:17:02.080467+0000","last_undegraded":"2024-04-15T15:17:02.080467+0000","last_fullsized":"2024-04-15T15:17:02.080467+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:42:24.767991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377736+0000","last_change":"2024-04-15T15:17:01.056955+0000","last_active":"2024-04-15T15:17:02.377736+0000","last_peered":"2024-04-15T15:17:02.377736+0000","last_clean":"2024-04-15T15:17:02.377736+0000","last_became_active":"2024-04-15T15:17:01.056604+0000","last_became_peered":"2024-04-15T15:17:01.056604+0000","last_unstale":"2024-04-15T15:17:02.377736+0000","last_undegraded":"2024-04-15T15:17:02.377736+0000","last_fullsized":"2024-04-15T15:17:02.377736+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080100+0000","last_change":"2024-04-15T15:17:01.064962+0000","last_active":"2024-04-15T15:17:02.080100+0000","last_peered":"2024-04-15T15:17:02.080100+0000","last_clean":"2024-04-15T15:17:02.080100+0000","last_became_active":"2024-04-15T15:17:01.064395+0000","last_became_peered":"2024-04-15T15:17:01.064395+0000","last_unstale":"2024-04-15T15:17:02.080100+0000","last_undegraded":"2024-04-15T15:17:02.080100+0000","last_fullsized":"2024-04-15T15:17:02.080100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:46.256635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063672+0000","last_change":"2024-04-15T15:17:01.069059+0000","last_active":"2024-04-15T15:17:02.063672+0000","last_peered":"2024-04-15T15:17:02.063672+0000","last_clean":"2024-04-15T15:17:02.063672+0000","last_became_active":"2024-04-15T15:17:01.067835+0000","last_became_peered":"2024-04-15T15:17:01.067835+0000","last_unstale":"2024-04-15T15:17:02.063672+0000","last_undegraded":"2024-04-15T15:17:02.063672+0000","last_fullsized":"2024-04-15T15:17:02.063672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:59.861915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080074+0000","last_change":"2024-04-15T15:17:01.065939+0000","last_active":"2024-04-15T15:17:02.080074+0000","last_peered":"2024-04-15T15:17:02.080074+0000","last_clean":"2024-04-15T15:17:02.080074+0000","last_became_active":"2024-04-15T15:17:01.065641+0000","last_became_peered":"2024-04-15T15:17:01.065641+0000","last_unstale":"2024-04-15T15:17:02.080074+0000","last_undegraded":"2024-04-15T15:17:02.080074+0000","last_fullsized":"2024-04-15T15:17:02.080074+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:25.547677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377846+0000","last_change":"2024-04-15T15:17:01.064428+0000","last_active":"2024-04-15T15:17:02.377846+0000","last_peered":"2024-04-15T15:17:02.377846+0000","last_clean":"2024-04-15T15:17:02.377846+0000","last_became_active":"2024-04-15T15:17:01.064147+0000","last_became_peered":"2024-04-15T15:17:01.064147+0000","last_unstale":"2024-04-15T15:17:02.377846+0000","last_undegraded":"2024-04-15T15:17:02.377846+0000","last_fullsized":"2024-04-15T15:17:02.377846+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378393+0000","last_change":"2024-04-15T15:17:01.067149+0000","last_active":"2024-04-15T15:17:02.378393+0000","last_peered":"2024-04-15T15:17:02.378393+0000","last_clean":"2024-04-15T15:17:02.378393+0000","last_became_active":"2024-04-15T15:17:01.066871+0000","last_became_peered":"2024-04-15T15:17:01.066871+0000","last_unstale":"2024-04-15T15:17:02.378393+0000","last_undegraded":"2024-04-15T15:17:02.378393+0000","last_fullsized":"2024-04-15T15:17:02.378393+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:24.305106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064304+0000","last_change":"2024-04-15T15:17:01.070470+0000","last_active":"2024-04-15T15:17:02.064304+0000","last_peered":"2024-04-15T15:17:02.064304+0000","last_clean":"2024-04-15T15:17:02.064304+0000","last_became_active":"2024-04-15T15:17:01.070200+0000","last_became_peered":"2024-04-15T15:17:01.070200+0000","last_unstale":"2024-04-15T15:17:02.064304+0000","last_undegraded":"2024-04-15T15:17:02.064304+0000","last_fullsized":"2024-04-15T15:17:02.064304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:59.861915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075348+0000","last_change":"2024-04-15T15:17:01.057828+0000","last_active":"2024-04-15T15:17:02.075348+0000","last_peered":"2024-04-15T15:17:02.075348+0000","last_clean":"2024-04-15T15:17:02.075348+0000","last_became_active":"2024-04-15T15:17:01.057532+0000","last_became_peered":"2024-04-15T15:17:01.057532+0000","last_unstale":"2024-04-15T15:17:02.075348+0000","last_undegraded":"2024-04-15T15:17:02.075348+0000","last_fullsized":"2024-04-15T15:17:02.075348+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:40:05.555933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064430+0000","last_change":"2024-04-15T15:17:01.071687+0000","last_active":"2024-04-15T15:17:02.064430+0000","last_peered":"2024-04-15T15:17:02.064430+0000","last_clean":"2024-04-15T15:17:02.064430+0000","last_became_active":"2024-04-15T15:17:01.071330+0000","last_became_peered":"2024-04-15T15:17:01.071330+0000","last_unstale":"2024-04-15T15:17:02.064430+0000","last_undegraded":"2024-04-15T15:17:02.064430+0000","last_fullsized":"2024-04-15T15:17:02.064430+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:23.644144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080121+0000","last_change":"2024-04-15T15:17:01.067759+0000","last_active":"2024-04-15T15:17:02.080121+0000","last_peered":"2024-04-15T15:17:02.080121+0000","last_clean":"2024-04-15T15:17:02.080121+0000","last_became_active":"2024-04-15T15:17:01.067476+0000","last_became_peered":"2024-04-15T15:17:01.067476+0000","last_unstale":"2024-04-15T15:17:02.080121+0000","last_undegraded":"2024-04-15T15:17:02.080121+0000","last_fullsized":"2024-04-15T15:17:02.080121+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:24.305106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064105+0000","last_change":"2024-04-15T15:17:01.070298+0000","last_active":"2024-04-15T15:17:02.064105+0000","last_peered":"2024-04-15T15:17:02.064105+0000","last_clean":"2024-04-15T15:17:02.064105+0000","last_became_active":"2024-04-15T15:17:01.070032+0000","last_became_peered":"2024-04-15T15:17:01.070032+0000","last_unstale":"2024-04-15T15:17:02.064105+0000","last_undegraded":"2024-04-15T15:17:02.064105+0000","last_fullsized":"2024-04-15T15:17:02.064105+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:02.918344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377955+0000","last_change":"2024-04-15T15:17:01.058285+0000","last_active":"2024-04-15T15:17:02.377955+0000","last_peered":"2024-04-15T15:17:02.377955+0000","last_clean":"2024-04-15T15:17:02.377955+0000","last_became_active":"2024-04-15T15:17:01.058001+0000","last_became_peered":"2024-04-15T15:17:01.058001+0000","last_unstale":"2024-04-15T15:17:02.377955+0000","last_undegraded":"2024-04-15T15:17:02.377955+0000","last_fullsized":"2024-04-15T15:17:02.377955+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075300+0000","last_change":"2024-04-15T15:17:01.064638+0000","last_active":"2024-04-15T15:17:02.075300+0000","last_peered":"2024-04-15T15:17:02.075300+0000","last_clean":"2024-04-15T15:17:02.075300+0000","last_became_active":"2024-04-15T15:17:01.064289+0000","last_became_peered":"2024-04-15T15:17:01.064289+0000","last_unstale":"2024-04-15T15:17:02.075300+0000","last_undegraded":"2024-04-15T15:17:02.075300+0000","last_fullsized":"2024-04-15T15:17:02.075300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:51.299505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063923+0000","last_change":"2024-04-15T15:17:01.056648+0000","last_active":"2024-04-15T15:17:02.063923+0000","last_peered":"2024-04-15T15:17:02.063923+0000","last_clean":"2024-04-15T15:17:02.063923+0000","last_became_active":"2024-04-15T15:17:01.056354+0000","last_became_peered":"2024-04-15T15:17:01.056354+0000","last_unstale":"2024-04-15T15:17:02.063923+0000","last_undegraded":"2024-04-15T15:17:02.063923+0000","last_fullsized":"2024-04-15T15:17:02.063923+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:23.034253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060738+0000","last_change":"2024-04-15T15:17:01.067781+0000","last_active":"2024-04-15T15:17:02.060738+0000","last_peered":"2024-04-15T15:17:02.060738+0000","last_clean":"2024-04-15T15:17:02.060738+0000","last_became_active":"2024-04-15T15:17:01.067517+0000","last_became_peered":"2024-04-15T15:17:01.067517+0000","last_unstale":"2024-04-15T15:17:02.060738+0000","last_undegraded":"2024-04-15T15:17:02.060738+0000","last_fullsized":"2024-04-15T15:17:02.060738+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063934+0000","last_change":"2024-04-15T15:17:01.059015+0000","last_active":"2024-04-15T15:17:02.063934+0000","last_peered":"2024-04-15T15:17:02.063934+0000","last_clean":"2024-04-15T15:17:02.063934+0000","last_became_active":"2024-04-15T15:17:01.058718+0000","last_became_peered":"2024-04-15T15:17:01.058718+0000","last_unstale":"2024-04-15T15:17:02.063934+0000","last_undegraded":"2024-04-15T15:17:02.063934+0000","last_fullsized":"2024-04-15T15:17:02.063934+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:04.108334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080356+0000","last_change":"2024-04-15T15:17:01.069694+0000","last_active":"2024-04-15T15:17:02.080356+0000","last_peered":"2024-04-15T15:17:02.080356+0000","last_clean":"2024-04-15T15:17:02.080356+0000","last_became_active":"2024-04-15T15:17:01.069408+0000","last_became_peered":"2024-04-15T15:17:01.069408+0000","last_unstale":"2024-04-15T15:17:02.080356+0000","last_undegraded":"2024-04-15T15:17:02.080356+0000","last_fullsized":"2024-04-15T15:17:02.080356+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:23.651512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064323+0000","last_change":"2024-04-15T15:17:01.068919+0000","last_active":"2024-04-15T15:17:02.064323+0000","last_peered":"2024-04-15T15:17:02.064323+0000","last_clean":"2024-04-15T15:17:02.064323+0000","last_became_active":"2024-04-15T15:17:01.068639+0000","last_became_peered":"2024-04-15T15:17:01.068639+0000","last_unstale":"2024-04-15T15:17:02.064323+0000","last_undegraded":"2024-04-15T15:17:02.064323+0000","last_fullsized":"2024-04-15T15:17:02.064323+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:46.277952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064537+0000","last_change":"2024-04-15T15:17:01.056050+0000","last_active":"2024-04-15T15:17:02.064537+0000","last_peered":"2024-04-15T15:17:02.064537+0000","last_clean":"2024-04-15T15:17:02.064537+0000","last_became_active":"2024-04-15T15:17:01.055735+0000","last_became_peered":"2024-04-15T15:17:01.055735+0000","last_unstale":"2024-04-15T15:17:02.064537+0000","last_undegraded":"2024-04-15T15:17:02.064537+0000","last_fullsized":"2024-04-15T15:17:02.064537+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:06.654184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378118+0000","last_change":"2024-04-15T15:17:01.058787+0000","last_active":"2024-04-15T15:17:02.378118+0000","last_peered":"2024-04-15T15:17:02.378118+0000","last_clean":"2024-04-15T15:17:02.378118+0000","last_became_active":"2024-04-15T15:17:01.058483+0000","last_became_peered":"2024-04-15T15:17:01.058483+0000","last_unstale":"2024-04-15T15:17:02.378118+0000","last_undegraded":"2024-04-15T15:17:02.378118+0000","last_fullsized":"2024-04-15T15:17:02.378118+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:38:18.560062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060461+0000","last_change":"2024-04-15T15:17:01.069926+0000","last_active":"2024-04-15T15:17:02.060461+0000","last_peered":"2024-04-15T15:17:02.060461+0000","last_clean":"2024-04-15T15:17:02.060461+0000","last_became_active":"2024-04-15T15:17:01.069632+0000","last_became_peered":"2024-04-15T15:17:01.069632+0000","last_unstale":"2024-04-15T15:17:02.060461+0000","last_undegraded":"2024-04-15T15:17:02.060461+0000","last_fullsized":"2024-04-15T15:17:02.060461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378199+0000","last_change":"2024-04-15T15:17:01.064698+0000","last_active":"2024-04-15T15:17:02.378199+0000","last_peered":"2024-04-15T15:17:02.378199+0000","last_clean":"2024-04-15T15:17:02.378199+0000","last_became_active":"2024-04-15T15:17:01.064241+0000","last_became_peered":"2024-04-15T15:17:01.064241+0000","last_unstale":"2024-04-15T15:17:02.378199+0000","last_undegraded":"2024-04-15T15:17:02.378199+0000","last_fullsized":"2024-04-15T15:17:02.378199+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:25.547677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064026+0000","last_change":"2024-04-15T15:17:00.053485+0000","last_active":"2024-04-15T15:17:02.064026+0000","last_peered":"2024-04-15T15:17:02.064026+0000","last_clean":"2024-04-15T15:17:02.064026+0000","last_became_active":"2024-04-15T15:17:00.053211+0000","last_became_peered":"2024-04-15T15:17:00.053211+0000","last_unstale":"2024-04-15T15:17:02.064026+0000","last_undegraded":"2024-04-15T15:17:02.064026+0000","last_fullsized":"2024-04-15T15:17:02.064026+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060777+0000","last_change":"2024-04-15T15:17:01.066600+0000","last_active":"2024-04-15T15:17:02.060777+0000","last_peered":"2024-04-15T15:17:02.060777+0000","last_clean":"2024-04-15T15:17:02.060777+0000","last_became_active":"2024-04-15T15:17:01.066291+0000","last_became_peered":"2024-04-15T15:17:01.066291+0000","last_unstale":"2024-04-15T15:17:02.060777+0000","last_undegraded":"2024-04-15T15:17:02.060777+0000","last_fullsized":"2024-04-15T15:17:02.060777+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:40:58.049667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377872+0000","last_change":"2024-04-15T15:17:00.056209+0000","last_active":"2024-04-15T15:17:02.377872+0000","last_peered":"2024-04-15T15:17:02.377872+0000","last_clean":"2024-04-15T15:17:02.377872+0000","last_became_active":"2024-04-15T15:17:00.055881+0000","last_became_peered":"2024-04-15T15:17:00.055881+0000","last_unstale":"2024-04-15T15:17:02.377872+0000","last_undegraded":"2024-04-15T15:17:02.377872+0000","last_fullsized":"2024-04-15T15:17:02.377872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064123+0000","last_change":"2024-04-15T15:17:01.067226+0000","last_active":"2024-04-15T15:17:02.064123+0000","last_peered":"2024-04-15T15:17:02.064123+0000","last_clean":"2024-04-15T15:17:02.064123+0000","last_became_active":"2024-04-15T15:17:01.066479+0000","last_became_peered":"2024-04-15T15:17:01.066479+0000","last_unstale":"2024-04-15T15:17:02.064123+0000","last_undegraded":"2024-04-15T15:17:02.064123+0000","last_fullsized":"2024-04-15T15:17:02.064123+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:37.773918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060829+0000","last_change":"2024-04-15T15:17:00.054251+0000","last_active":"2024-04-15T15:17:02.060829+0000","last_peered":"2024-04-15T15:17:02.060829+0000","last_clean":"2024-04-15T15:17:02.060829+0000","last_became_active":"2024-04-15T15:17:00.053980+0000","last_became_peered":"2024-04-15T15:17:00.053980+0000","last_unstale":"2024-04-15T15:17:02.060829+0000","last_undegraded":"2024-04-15T15:17:02.060829+0000","last_fullsized":"2024-04-15T15:17:02.060829+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:23.176250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378093+0000","last_change":"2024-04-15T15:17:01.062667+0000","last_active":"2024-04-15T15:17:02.378093+0000","last_peered":"2024-04-15T15:17:02.378093+0000","last_clean":"2024-04-15T15:17:02.378093+0000","last_became_active":"2024-04-15T15:17:01.062171+0000","last_became_peered":"2024-04-15T15:17:01.062171+0000","last_unstale":"2024-04-15T15:17:02.378093+0000","last_undegraded":"2024-04-15T15:17:02.378093+0000","last_fullsized":"2024-04-15T15:17:02.378093+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:23.651512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064694+0000","last_change":"2024-04-15T15:17:00.058378+0000","last_active":"2024-04-15T15:17:02.064694+0000","last_peered":"2024-04-15T15:17:02.064694+0000","last_clean":"2024-04-15T15:17:02.064694+0000","last_became_active":"2024-04-15T15:17:00.058024+0000","last_became_peered":"2024-04-15T15:17:00.058024+0000","last_unstale":"2024-04-15T15:17:02.064694+0000","last_undegraded":"2024-04-15T15:17:02.064694+0000","last_fullsized":"2024-04-15T15:17:02.064694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:10:50.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377353+0000","last_change":"2024-04-15T15:17:01.071377+0000","last_active":"2024-04-15T15:17:02.377353+0000","last_peered":"2024-04-15T15:17:02.377353+0000","last_clean":"2024-04-15T15:17:02.377353+0000","last_became_active":"2024-04-15T15:17:01.071087+0000","last_became_peered":"2024-04-15T15:17:01.071087+0000","last_unstale":"2024-04-15T15:17:02.377353+0000","last_undegraded":"2024-04-15T15:17:02.377353+0000","last_fullsized":"2024-04-15T15:17:02.377353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080210+0000","last_change":"2024-04-15T15:17:00.052588+0000","last_active":"2024-04-15T15:17:02.080210+0000","last_peered":"2024-04-15T15:17:02.080210+0000","last_clean":"2024-04-15T15:17:02.080210+0000","last_became_active":"2024-04-15T15:17:00.052287+0000","last_became_peered":"2024-04-15T15:17:00.052287+0000","last_unstale":"2024-04-15T15:17:02.080210+0000","last_undegraded":"2024-04-15T15:17:02.080210+0000","last_fullsized":"2024-04-15T15:17:02.080210+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080214+0000","last_change":"2024-04-15T15:17:01.077529+0000","last_active":"2024-04-15T15:17:02.080214+0000","last_peered":"2024-04-15T15:17:02.080214+0000","last_clean":"2024-04-15T15:17:02.080214+0000","last_became_active":"2024-04-15T15:17:01.077231+0000","last_became_peered":"2024-04-15T15:17:01.077231+0000","last_unstale":"2024-04-15T15:17:02.080214+0000","last_undegraded":"2024-04-15T15:17:02.080214+0000","last_fullsized":"2024-04-15T15:17:02.080214+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:19:19.245967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378089+0000","last_change":"2024-04-15T15:17:00.060924+0000","last_active":"2024-04-15T15:17:02.378089+0000","last_peered":"2024-04-15T15:17:02.378089+0000","last_clean":"2024-04-15T15:17:02.378089+0000","last_became_active":"2024-04-15T15:17:00.060610+0000","last_became_peered":"2024-04-15T15:17:00.060610+0000","last_unstale":"2024-04-15T15:17:02.378089+0000","last_undegraded":"2024-04-15T15:17:02.378089+0000","last_fullsized":"2024-04-15T15:17:02.378089+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:35:23.176250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064248+0000","last_change":"2024-04-15T15:17:01.068809+0000","last_active":"2024-04-15T15:17:02.064248+0000","last_peered":"2024-04-15T15:17:02.064248+0000","last_clean":"2024-04-15T15:17:02.064248+0000","last_became_active":"2024-04-15T15:17:01.067740+0000","last_became_peered":"2024-04-15T15:17:01.067740+0000","last_unstale":"2024-04-15T15:17:02.064248+0000","last_undegraded":"2024-04-15T15:17:02.064248+0000","last_fullsized":"2024-04-15T15:17:02.064248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:00:50.094085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060464+0000","last_change":"2024-04-15T15:17:00.056079+0000","last_active":"2024-04-15T15:17:02.060464+0000","last_peered":"2024-04-15T15:17:02.060464+0000","last_clean":"2024-04-15T15:17:02.060464+0000","last_became_active":"2024-04-15T15:17:00.055794+0000","last_became_peered":"2024-04-15T15:17:00.055794+0000","last_unstale":"2024-04-15T15:17:02.060464+0000","last_undegraded":"2024-04-15T15:17:02.060464+0000","last_fullsized":"2024-04-15T15:17:02.060464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:56.456359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377746+0000","last_change":"2024-04-15T15:17:01.071336+0000","last_active":"2024-04-15T15:17:02.377746+0000","last_peered":"2024-04-15T15:17:02.377746+0000","last_clean":"2024-04-15T15:17:02.377746+0000","last_became_active":"2024-04-15T15:17:01.070764+0000","last_became_peered":"2024-04-15T15:17:01.070764+0000","last_unstale":"2024-04-15T15:17:02.377746+0000","last_undegraded":"2024-04-15T15:17:02.377746+0000","last_fullsized":"2024-04-15T15:17:02.377746+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:24.305106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063479+0000","last_change":"2024-04-15T15:17:00.048501+0000","last_active":"2024-04-15T15:17:02.063479+0000","last_peered":"2024-04-15T15:17:02.063479+0000","last_clean":"2024-04-15T15:17:02.063479+0000","last_became_active":"2024-04-15T15:17:00.048148+0000","last_became_peered":"2024-04-15T15:17:00.048148+0000","last_unstale":"2024-04-15T15:17:02.063479+0000","last_undegraded":"2024-04-15T15:17:02.063479+0000","last_fullsized":"2024-04-15T15:17:02.063479+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:31:58.375472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075491+0000","last_change":"2024-04-15T15:17:01.071813+0000","last_active":"2024-04-15T15:17:02.075491+0000","last_peered":"2024-04-15T15:17:02.075491+0000","last_clean":"2024-04-15T15:17:02.075491+0000","last_became_active":"2024-04-15T15:17:01.071521+0000","last_became_peered":"2024-04-15T15:17:01.071521+0000","last_unstale":"2024-04-15T15:17:02.075491+0000","last_undegraded":"2024-04-15T15:17:02.075491+0000","last_fullsized":"2024-04-15T15:17:02.075491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:20.614860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378154+0000","last_change":"2024-04-15T15:17:00.059059+0000","last_active":"2024-04-15T15:17:02.378154+0000","last_peered":"2024-04-15T15:17:02.378154+0000","last_clean":"2024-04-15T15:17:02.378154+0000","last_became_active":"2024-04-15T15:17:00.058527+0000","last_became_peered":"2024-04-15T15:17:00.058527+0000","last_unstale":"2024-04-15T15:17:02.378154+0000","last_undegraded":"2024-04-15T15:17:02.378154+0000","last_fullsized":"2024-04-15T15:17:02.378154+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378300+0000","last_change":"2024-04-15T15:17:01.063386+0000","last_active":"2024-04-15T15:17:02.378300+0000","last_peered":"2024-04-15T15:17:02.378300+0000","last_clean":"2024-04-15T15:17:02.378300+0000","last_became_active":"2024-04-15T15:17:01.062362+0000","last_became_peered":"2024-04-15T15:17:01.062362+0000","last_unstale":"2024-04-15T15:17:02.378300+0000","last_undegraded":"2024-04-15T15:17:02.378300+0000","last_fullsized":"2024-04-15T15:17:02.378300+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:46.256635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080573+0000","last_change":"2024-04-15T15:17:00.058594+0000","last_active":"2024-04-15T15:17:02.080573+0000","last_peered":"2024-04-15T15:17:02.080573+0000","last_clean":"2024-04-15T15:17:02.080573+0000","last_became_active":"2024-04-15T15:17:00.058359+0000","last_became_peered":"2024-04-15T15:17:00.058359+0000","last_unstale":"2024-04-15T15:17:02.080573+0000","last_undegraded":"2024-04-15T15:17:02.080573+0000","last_fullsized":"2024-04-15T15:17:02.080573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:45.273128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377781+0000","last_change":"2024-04-15T15:17:01.065464+0000","last_active":"2024-04-15T15:17:02.377781+0000","last_peered":"2024-04-15T15:17:02.377781+0000","last_clean":"2024-04-15T15:17:02.377781+0000","last_became_active":"2024-04-15T15:17:01.065142+0000","last_became_peered":"2024-04-15T15:17:01.065142+0000","last_unstale":"2024-04-15T15:17:02.377781+0000","last_undegraded":"2024-04-15T15:17:02.377781+0000","last_fullsized":"2024-04-15T15:17:02.377781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:30:58.920688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080574+0000","last_change":"2024-04-15T15:17:00.055108+0000","last_active":"2024-04-15T15:17:02.080574+0000","last_peered":"2024-04-15T15:17:02.080574+0000","last_clean":"2024-04-15T15:17:02.080574+0000","last_became_active":"2024-04-15T15:17:00.054818+0000","last_became_peered":"2024-04-15T15:17:00.054818+0000","last_unstale":"2024-04-15T15:17:02.080574+0000","last_undegraded":"2024-04-15T15:17:02.080574+0000","last_fullsized":"2024-04-15T15:17:02.080574+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064040+0000","last_change":"2024-04-15T15:17:01.066511+0000","last_active":"2024-04-15T15:17:02.064040+0000","last_peered":"2024-04-15T15:17:02.064040+0000","last_clean":"2024-04-15T15:17:02.064040+0000","last_became_active":"2024-04-15T15:17:01.066201+0000","last_became_peered":"2024-04-15T15:17:01.066201+0000","last_unstale":"2024-04-15T15:17:02.064040+0000","last_undegraded":"2024-04-15T15:17:02.064040+0000","last_fullsized":"2024-04-15T15:17:02.064040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:06.654184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377819+0000","last_change":"2024-04-15T15:17:00.061109+0000","last_active":"2024-04-15T15:17:02.377819+0000","last_peered":"2024-04-15T15:17:02.377819+0000","last_clean":"2024-04-15T15:17:02.377819+0000","last_became_active":"2024-04-15T15:17:00.060823+0000","last_became_peered":"2024-04-15T15:17:00.060823+0000","last_unstale":"2024-04-15T15:17:02.377819+0000","last_undegraded":"2024-04-15T15:17:02.377819+0000","last_fullsized":"2024-04-15T15:17:02.377819+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:58:36.769094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377423+0000","last_change":"2024-04-15T15:17:01.067591+0000","last_active":"2024-04-15T15:17:02.377423+0000","last_peered":"2024-04-15T15:17:02.377423+0000","last_clean":"2024-04-15T15:17:02.377423+0000","last_became_active":"2024-04-15T15:17:01.065638+0000","last_became_peered":"2024-04-15T15:17:01.065638+0000","last_unstale":"2024-04-15T15:17:02.377423+0000","last_undegraded":"2024-04-15T15:17:02.377423+0000","last_fullsized":"2024-04-15T15:17:02.377423+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377974+0000","last_change":"2024-04-15T15:17:00.057661+0000","last_active":"2024-04-15T15:17:02.377974+0000","last_peered":"2024-04-15T15:17:02.377974+0000","last_clean":"2024-04-15T15:17:02.377974+0000","last_became_active":"2024-04-15T15:17:00.057376+0000","last_became_peered":"2024-04-15T15:17:00.057376+0000","last_unstale":"2024-04-15T15:17:02.377974+0000","last_undegraded":"2024-04-15T15:17:02.377974+0000","last_fullsized":"2024-04-15T15:17:02.377974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:22.639320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377664+0000","last_change":"2024-04-15T15:17:01.061592+0000","last_active":"2024-04-15T15:17:02.377664+0000","last_peered":"2024-04-15T15:17:02.377664+0000","last_clean":"2024-04-15T15:17:02.377664+0000","last_became_active":"2024-04-15T15:17:01.061250+0000","last_became_peered":"2024-04-15T15:17:01.061250+0000","last_unstale":"2024-04-15T15:17:02.377664+0000","last_undegraded":"2024-04-15T15:17:02.377664+0000","last_fullsized":"2024-04-15T15:17:02.377664+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:46.256635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060563+0000","last_change":"2024-04-15T15:17:00.055785+0000","last_active":"2024-04-15T15:17:02.060563+0000","last_peered":"2024-04-15T15:17:02.060563+0000","last_clean":"2024-04-15T15:17:02.060563+0000","last_became_active":"2024-04-15T15:17:00.055509+0000","last_became_peered":"2024-04-15T15:17:00.055509+0000","last_unstale":"2024-04-15T15:17:02.060563+0000","last_undegraded":"2024-04-15T15:17:02.060563+0000","last_fullsized":"2024-04-15T15:17:02.060563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:06:19.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377999+0000","last_change":"2024-04-15T15:17:01.056835+0000","last_active":"2024-04-15T15:17:02.377999+0000","last_peered":"2024-04-15T15:17:02.377999+0000","last_clean":"2024-04-15T15:17:02.377999+0000","last_became_active":"2024-04-15T15:17:01.056566+0000","last_became_peered":"2024-04-15T15:17:01.056566+0000","last_unstale":"2024-04-15T15:17:02.377999+0000","last_undegraded":"2024-04-15T15:17:02.377999+0000","last_fullsized":"2024-04-15T15:17:02.377999+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:27.121130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063397+0000","last_change":"2024-04-15T15:17:00.055842+0000","last_active":"2024-04-15T15:17:02.063397+0000","last_peered":"2024-04-15T15:17:02.063397+0000","last_clean":"2024-04-15T15:17:02.063397+0000","last_became_active":"2024-04-15T15:17:00.054053+0000","last_became_peered":"2024-04-15T15:17:00.054053+0000","last_unstale":"2024-04-15T15:17:02.063397+0000","last_undegraded":"2024-04-15T15:17:02.063397+0000","last_fullsized":"2024-04-15T15:17:02.063397+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377880+0000","last_change":"2024-04-15T15:17:01.060443+0000","last_active":"2024-04-15T15:17:02.377880+0000","last_peered":"2024-04-15T15:17:02.377880+0000","last_clean":"2024-04-15T15:17:02.377880+0000","last_became_active":"2024-04-15T15:17:01.060102+0000","last_became_peered":"2024-04-15T15:17:01.060102+0000","last_unstale":"2024-04-15T15:17:02.377880+0000","last_undegraded":"2024-04-15T15:17:02.377880+0000","last_fullsized":"2024-04-15T15:17:02.377880+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:08.601149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377886+0000","last_change":"2024-04-15T15:17:00.061010+0000","last_active":"2024-04-15T15:17:02.377886+0000","last_peered":"2024-04-15T15:17:02.377886+0000","last_clean":"2024-04-15T15:17:02.377886+0000","last_became_active":"2024-04-15T15:17:00.060723+0000","last_became_peered":"2024-04-15T15:17:00.060723+0000","last_unstale":"2024-04-15T15:17:02.377886+0000","last_undegraded":"2024-04-15T15:17:02.377886+0000","last_fullsized":"2024-04-15T15:17:02.377886+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:22.029429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377635+0000","last_change":"2024-04-15T15:17:01.060543+0000","last_active":"2024-04-15T15:17:02.377635+0000","last_peered":"2024-04-15T15:17:02.377635+0000","last_clean":"2024-04-15T15:17:02.377635+0000","last_became_active":"2024-04-15T15:17:01.060255+0000","last_became_peered":"2024-04-15T15:17:01.060255+0000","last_unstale":"2024-04-15T15:17:02.377635+0000","last_undegraded":"2024-04-15T15:17:02.377635+0000","last_fullsized":"2024-04-15T15:17:02.377635+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:47.544340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080591+0000","last_change":"2024-04-15T15:17:00.053088+0000","last_active":"2024-04-15T15:17:02.080591+0000","last_peered":"2024-04-15T15:17:02.080591+0000","last_clean":"2024-04-15T15:17:02.080591+0000","last_became_active":"2024-04-15T15:17:00.052762+0000","last_became_peered":"2024-04-15T15:17:00.052762+0000","last_unstale":"2024-04-15T15:17:02.080591+0000","last_undegraded":"2024-04-15T15:17:02.080591+0000","last_fullsized":"2024-04-15T15:17:02.080591+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080469+0000","last_change":"2024-04-15T15:17:01.067435+0000","last_active":"2024-04-15T15:17:02.080469+0000","last_peered":"2024-04-15T15:17:02.080469+0000","last_clean":"2024-04-15T15:17:02.080469+0000","last_became_active":"2024-04-15T15:17:01.066970+0000","last_became_peered":"2024-04-15T15:17:01.066970+0000","last_unstale":"2024-04-15T15:17:02.080469+0000","last_undegraded":"2024-04-15T15:17:02.080469+0000","last_fullsized":"2024-04-15T15:17:02.080469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:09:44.232483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063970+0000","last_change":"2024-04-15T15:17:00.049730+0000","last_active":"2024-04-15T15:17:02.063970+0000","last_peered":"2024-04-15T15:17:02.063970+0000","last_clean":"2024-04-15T15:17:02.063970+0000","last_became_active":"2024-04-15T15:17:00.049472+0000","last_became_peered":"2024-04-15T15:17:00.049472+0000","last_unstale":"2024-04-15T15:17:02.063970+0000","last_undegraded":"2024-04-15T15:17:02.063970+0000","last_fullsized":"2024-04-15T15:17:02.063970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063849+0000","last_change":"2024-04-15T15:17:01.067474+0000","last_active":"2024-04-15T15:17:02.063849+0000","last_peered":"2024-04-15T15:17:02.063849+0000","last_clean":"2024-04-15T15:17:02.063849+0000","last_became_active":"2024-04-15T15:17:01.066802+0000","last_became_peered":"2024-04-15T15:17:01.066802+0000","last_unstale":"2024-04-15T15:17:02.063849+0000","last_undegraded":"2024-04-15T15:17:02.063849+0000","last_fullsized":"2024-04-15T15:17:02.063849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:46.277952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075391+0000","last_change":"2024-04-15T15:17:00.054252+0000","last_active":"2024-04-15T15:17:02.075391+0000","last_peered":"2024-04-15T15:17:02.075391+0000","last_clean":"2024-04-15T15:17:02.075391+0000","last_became_active":"2024-04-15T15:17:00.053917+0000","last_became_peered":"2024-04-15T15:17:00.053917+0000","last_unstale":"2024-04-15T15:17:02.075391+0000","last_undegraded":"2024-04-15T15:17:02.075391+0000","last_fullsized":"2024-04-15T15:17:02.075391+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:24:18.668440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075557+0000","last_change":"2024-04-15T15:17:01.058076+0000","last_active":"2024-04-15T15:17:02.075557+0000","last_peered":"2024-04-15T15:17:02.075557+0000","last_clean":"2024-04-15T15:17:02.075557+0000","last_became_active":"2024-04-15T15:17:01.057615+0000","last_became_peered":"2024-04-15T15:17:01.057615+0000","last_unstale":"2024-04-15T15:17:02.075557+0000","last_undegraded":"2024-04-15T15:17:02.075557+0000","last_fullsized":"2024-04-15T15:17:02.075557+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:57.461183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060631+0000","last_change":"2024-04-15T15:17:00.053573+0000","last_active":"2024-04-15T15:17:02.060631+0000","last_peered":"2024-04-15T15:17:02.060631+0000","last_clean":"2024-04-15T15:17:02.060631+0000","last_became_active":"2024-04-15T15:17:00.053296+0000","last_became_peered":"2024-04-15T15:17:00.053296+0000","last_unstale":"2024-04-15T15:17:02.060631+0000","last_undegraded":"2024-04-15T15:17:02.060631+0000","last_fullsized":"2024-04-15T15:17:02.060631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378107+0000","last_change":"2024-04-15T15:17:01.061605+0000","last_active":"2024-04-15T15:17:02.378107+0000","last_peered":"2024-04-15T15:17:02.378107+0000","last_clean":"2024-04-15T15:17:02.378107+0000","last_became_active":"2024-04-15T15:17:01.061267+0000","last_became_peered":"2024-04-15T15:17:01.061267+0000","last_unstale":"2024-04-15T15:17:02.378107+0000","last_undegraded":"2024-04-15T15:17:02.378107+0000","last_fullsized":"2024-04-15T15:17:02.378107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:58:51.072457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075228+0000","last_change":"2024-04-15T15:17:00.052685+0000","last_active":"2024-04-15T15:17:02.075228+0000","last_peered":"2024-04-15T15:17:02.075228+0000","last_clean":"2024-04-15T15:17:02.075228+0000","last_became_active":"2024-04-15T15:17:00.052398+0000","last_became_peered":"2024-04-15T15:17:00.052398+0000","last_unstale":"2024-04-15T15:17:02.075228+0000","last_undegraded":"2024-04-15T15:17:02.075228+0000","last_fullsized":"2024-04-15T15:17:02.075228+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:30:51.941942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377775+0000","last_change":"2024-04-15T15:17:01.064764+0000","last_active":"2024-04-15T15:17:02.377775+0000","last_peered":"2024-04-15T15:17:02.377775+0000","last_clean":"2024-04-15T15:17:02.377775+0000","last_became_active":"2024-04-15T15:17:01.064496+0000","last_became_peered":"2024-04-15T15:17:01.064496+0000","last_unstale":"2024-04-15T15:17:02.377775+0000","last_undegraded":"2024-04-15T15:17:02.377775+0000","last_fullsized":"2024-04-15T15:17:02.377775+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:40:24.356651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075244+0000","last_change":"2024-04-15T15:17:00.044538+0000","last_active":"2024-04-15T15:17:02.075244+0000","last_peered":"2024-04-15T15:17:02.075244+0000","last_clean":"2024-04-15T15:17:02.075244+0000","last_became_active":"2024-04-15T15:17:00.044237+0000","last_became_peered":"2024-04-15T15:17:00.044237+0000","last_unstale":"2024-04-15T15:17:02.075244+0000","last_undegraded":"2024-04-15T15:17:02.075244+0000","last_fullsized":"2024-04-15T15:17:02.075244+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:16:18.037173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080194+0000","last_change":"2024-04-15T15:17:01.066478+0000","last_active":"2024-04-15T15:17:02.080194+0000","last_peered":"2024-04-15T15:17:02.080194+0000","last_clean":"2024-04-15T15:17:02.080194+0000","last_became_active":"2024-04-15T15:17:01.066229+0000","last_became_peered":"2024-04-15T15:17:01.066229+0000","last_unstale":"2024-04-15T15:17:02.080194+0000","last_undegraded":"2024-04-15T15:17:02.080194+0000","last_fullsized":"2024-04-15T15:17:02.080194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:08.601149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378194+0000","last_change":"2024-04-15T15:17:00.064123+0000","last_active":"2024-04-15T15:17:02.378194+0000","last_peered":"2024-04-15T15:17:02.378194+0000","last_clean":"2024-04-15T15:17:02.378194+0000","last_became_active":"2024-04-15T15:17:00.063535+0000","last_became_peered":"2024-04-15T15:17:00.063535+0000","last_unstale":"2024-04-15T15:17:02.378194+0000","last_undegraded":"2024-04-15T15:17:02.378194+0000","last_fullsized":"2024-04-15T15:17:02.378194+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:58.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378009+0000","last_change":"2024-04-15T15:17:01.071081+0000","last_active":"2024-04-15T15:17:02.378009+0000","last_peered":"2024-04-15T15:17:02.378009+0000","last_clean":"2024-04-15T15:17:02.378009+0000","last_became_active":"2024-04-15T15:17:01.070378+0000","last_became_peered":"2024-04-15T15:17:01.070378+0000","last_unstale":"2024-04-15T15:17:02.378009+0000","last_undegraded":"2024-04-15T15:17:02.378009+0000","last_fullsized":"2024-04-15T15:17:02.378009+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:19:19.245967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075171+0000","last_change":"2024-04-15T15:17:00.051113+0000","last_active":"2024-04-15T15:17:02.075171+0000","last_peered":"2024-04-15T15:17:02.075171+0000","last_clean":"2024-04-15T15:17:02.075171+0000","last_became_active":"2024-04-15T15:17:00.050824+0000","last_became_peered":"2024-04-15T15:17:00.050824+0000","last_unstale":"2024-04-15T15:17:02.075171+0000","last_undegraded":"2024-04-15T15:17:02.075171+0000","last_fullsized":"2024-04-15T15:17:02.075171+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:35:25.744871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060631+0000","last_change":"2024-04-15T15:17:01.067058+0000","last_active":"2024-04-15T15:17:02.060631+0000","last_peered":"2024-04-15T15:17:02.060631+0000","last_clean":"2024-04-15T15:17:02.060631+0000","last_became_active":"2024-04-15T15:17:01.066787+0000","last_became_peered":"2024-04-15T15:17:01.066787+0000","last_unstale":"2024-04-15T15:17:02.060631+0000","last_undegraded":"2024-04-15T15:17:02.060631+0000","last_fullsized":"2024-04-15T15:17:02.060631+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:39.912600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080684+0000","last_change":"2024-04-15T15:17:00.058333+0000","last_active":"2024-04-15T15:17:02.080684+0000","last_peered":"2024-04-15T15:17:02.080684+0000","last_clean":"2024-04-15T15:17:02.080684+0000","last_became_active":"2024-04-15T15:17:00.057892+0000","last_became_peered":"2024-04-15T15:17:00.057892+0000","last_unstale":"2024-04-15T15:17:02.080684+0000","last_undegraded":"2024-04-15T15:17:02.080684+0000","last_fullsized":"2024-04-15T15:17:02.080684+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:05:01.913520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378605+0000","last_change":"2024-04-15T15:17:01.062606+0000","last_active":"2024-04-15T15:17:02.378605+0000","last_peered":"2024-04-15T15:17:02.378605+0000","last_clean":"2024-04-15T15:17:02.378605+0000","last_became_active":"2024-04-15T15:17:01.062062+0000","last_became_peered":"2024-04-15T15:17:01.062062+0000","last_unstale":"2024-04-15T15:17:02.378605+0000","last_undegraded":"2024-04-15T15:17:02.378605+0000","last_fullsized":"2024-04-15T15:17:02.378605+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:14:08.601149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.075486+0000","last_change":"2024-04-15T15:17:00.049576+0000","last_active":"2024-04-15T15:17:02.075486+0000","last_peered":"2024-04-15T15:17:02.075486+0000","last_clean":"2024-04-15T15:17:02.075486+0000","last_became_active":"2024-04-15T15:17:00.049280+0000","last_became_peered":"2024-04-15T15:17:00.049280+0000","last_unstale":"2024-04-15T15:17:02.075486+0000","last_undegraded":"2024-04-15T15:17:02.075486+0000","last_fullsized":"2024-04-15T15:17:02.075486+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:15:28.100508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378101+0000","last_change":"2024-04-15T15:17:01.069695+0000","last_active":"2024-04-15T15:17:02.378101+0000","last_peered":"2024-04-15T15:17:02.378101+0000","last_clean":"2024-04-15T15:17:02.378101+0000","last_became_active":"2024-04-15T15:17:01.069323+0000","last_became_peered":"2024-04-15T15:17:01.069323+0000","last_unstale":"2024-04-15T15:17:02.378101+0000","last_undegraded":"2024-04-15T15:17:02.378101+0000","last_fullsized":"2024-04-15T15:17:02.378101+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:42:24.767991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377573+0000","last_change":"2024-04-15T15:17:00.061061+0000","last_active":"2024-04-15T15:17:02.377573+0000","last_peered":"2024-04-15T15:17:02.377573+0000","last_clean":"2024-04-15T15:17:02.377573+0000","last_became_active":"2024-04-15T15:17:00.060731+0000","last_became_peered":"2024-04-15T15:17:00.060731+0000","last_unstale":"2024-04-15T15:17:02.377573+0000","last_undegraded":"2024-04-15T15:17:02.377573+0000","last_fullsized":"2024-04-15T15:17:02.377573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:56.456359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377520+0000","last_change":"2024-04-15T15:17:01.065896+0000","last_active":"2024-04-15T15:17:02.377520+0000","last_peered":"2024-04-15T15:17:02.377520+0000","last_clean":"2024-04-15T15:17:02.377520+0000","last_became_active":"2024-04-15T15:17:01.065259+0000","last_became_peered":"2024-04-15T15:17:01.065259+0000","last_unstale":"2024-04-15T15:17:02.377520+0000","last_undegraded":"2024-04-15T15:17:02.377520+0000","last_fullsized":"2024-04-15T15:17:02.377520+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:26:31.443774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377628+0000","last_change":"2024-04-15T15:17:00.063764+0000","last_active":"2024-04-15T15:17:02.377628+0000","last_peered":"2024-04-15T15:17:02.377628+0000","last_clean":"2024-04-15T15:17:02.377628+0000","last_became_active":"2024-04-15T15:17:00.063528+0000","last_became_peered":"2024-04-15T15:17:00.063528+0000","last_unstale":"2024-04-15T15:17:02.377628+0000","last_undegraded":"2024-04-15T15:17:02.377628+0000","last_fullsized":"2024-04-15T15:17:02.377628+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:17.256260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.060612+0000","last_change":"2024-04-15T15:17:01.070588+0000","last_active":"2024-04-15T15:17:02.060612+0000","last_peered":"2024-04-15T15:17:02.060612+0000","last_clean":"2024-04-15T15:17:02.060612+0000","last_became_active":"2024-04-15T15:17:01.070263+0000","last_became_peered":"2024-04-15T15:17:01.070263+0000","last_unstale":"2024-04-15T15:17:02.060612+0000","last_undegraded":"2024-04-15T15:17:02.060612+0000","last_fullsized":"2024-04-15T15:17:02.060612+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:04:04.025731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.064627+0000","last_change":"2024-04-15T15:17:00.056627+0000","last_active":"2024-04-15T15:17:02.064627+0000","last_peered":"2024-04-15T15:17:02.064627+0000","last_clean":"2024-04-15T15:17:02.064627+0000","last_became_active":"2024-04-15T15:17:00.056317+0000","last_became_peered":"2024-04-15T15:17:00.056317+0000","last_unstale":"2024-04-15T15:17:02.064627+0000","last_undegraded":"2024-04-15T15:17:02.064627+0000","last_fullsized":"2024-04-15T15:17:02.064627+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:25:32.018397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063854+0000","last_change":"2024-04-15T15:17:01.071745+0000","last_active":"2024-04-15T15:17:02.063854+0000","last_peered":"2024-04-15T15:17:02.063854+0000","last_clean":"2024-04-15T15:17:02.063854+0000","last_became_active":"2024-04-15T15:17:01.071466+0000","last_became_peered":"2024-04-15T15:17:01.071466+0000","last_unstale":"2024-04-15T15:17:02.063854+0000","last_undegraded":"2024-04-15T15:17:02.063854+0000","last_fullsized":"2024-04-15T15:17:02.063854+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:57:23.644144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.063563+0000","last_change":"2024-04-15T15:17:00.057863+0000","last_active":"2024-04-15T15:17:02.063563+0000","last_peered":"2024-04-15T15:17:02.063563+0000","last_clean":"2024-04-15T15:17:02.063563+0000","last_became_active":"2024-04-15T15:17:00.057596+0000","last_became_peered":"2024-04-15T15:17:00.057596+0000","last_unstale":"2024-04-15T15:17:02.063563+0000","last_undegraded":"2024-04-15T15:17:02.063563+0000","last_fullsized":"2024-04-15T15:17:02.063563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:51:20.085875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378193+0000","last_change":"2024-04-15T15:17:01.060073+0000","last_active":"2024-04-15T15:17:02.378193+0000","last_peered":"2024-04-15T15:17:02.378193+0000","last_clean":"2024-04-15T15:17:02.378193+0000","last_became_active":"2024-04-15T15:17:01.059802+0000","last_became_peered":"2024-04-15T15:17:01.059802+0000","last_unstale":"2024-04-15T15:17:02.378193+0000","last_undegraded":"2024-04-15T15:17:02.378193+0000","last_fullsized":"2024-04-15T15:17:02.378193+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:39:47.544340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.377563+0000","last_change":"2024-04-15T15:17:00.058811+0000","last_active":"2024-04-15T15:17:02.377563+0000","last_peered":"2024-04-15T15:17:02.377563+0000","last_clean":"2024-04-15T15:17:02.377563+0000","last_became_active":"2024-04-15T15:17:00.058530+0000","last_became_peered":"2024-04-15T15:17:00.058530+0000","last_unstale":"2024-04-15T15:17:02.377563+0000","last_undegraded":"2024-04-15T15:17:02.377563+0000","last_fullsized":"2024-04-15T15:17:02.377563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:40:05.649360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.080486+0000","last_change":"2024-04-15T15:17:01.070460+0000","last_active":"2024-04-15T15:17:02.080486+0000","last_peered":"2024-04-15T15:17:02.080486+0000","last_clean":"2024-04-15T15:17:02.080486+0000","last_became_active":"2024-04-15T15:17:01.070110+0000","last_became_peered":"2024-04-15T15:17:01.070110+0000","last_unstale":"2024-04-15T15:17:02.080486+0000","last_undegraded":"2024-04-15T15:17:02.080486+0000","last_fullsized":"2024-04-15T15:17:02.080486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:17:00.033889+0000","last_clean_scrub_stamp":"2024-04-15T15:17:00.033889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:18.093938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378282+0000","last_change":"2024-04-15T15:17:00.061284+0000","last_active":"2024-04-15T15:17:02.378282+0000","last_peered":"2024-04-15T15:17:02.378282+0000","last_clean":"2024-04-15T15:17:02.378282+0000","last_became_active":"2024-04-15T15:17:00.060520+0000","last_became_peered":"2024-04-15T15:17:00.060520+0000","last_unstale":"2024-04-15T15:17:02.378282+0000","last_undegraded":"2024-04-15T15:17:02.378282+0000","last_fullsized":"2024-04-15T15:17:02.378282+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:59.029065+0000","last_clean_scrub_stamp":"2024-04-15T15:16:59.029065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:18:03.103510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":202,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:17:02.378054+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:02.378054+0000","last_peered":"2024-04-15T15:17:02.378054+0000","last_clean":"2024-04-15T15:17:02.378054+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:02.378054+0000","last_undegraded":"2024-04-15T15:17:02.378054+0000","last_fullsized":"2024-04-15T15:17:02.378054+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":450560,"data_stored":194433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542168,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":193963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":194455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1056768,"data_stored":785901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":462848,"data_stored":195497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":450560,"data_stored":193921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":450560,"data_stored":193941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339640,"statfs":{"total":96636764160,"available":96603791360,"internally_reserved":0,"allocated":1044480,"data_stored":784253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:09.899 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:17:09.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-15T15:17:09.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:09.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.042+0000 7fc3b2d74640 1 Processor -- start 2024-04-15T15:17:10.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.042+0000 7fc3b2d74640 1 -- start start 2024-04-15T15:17:10.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b2d74640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3ac15ecb0 0x7fc3ac0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b2d74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac0af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b2d74640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3ac0b10d0 con 0x7fc3ac0b4350 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b2d74640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc3ac15f090 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b2d74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ac0cdf80 con 0x7fc3ac0b1d30 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b1311640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.043+0000 7fc3b0b10640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3ac15ecb0 0x7fc3ac0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3b0b10640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3ac15ecb0 0x7fc3ac0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:55000/0 (socket says 172.21.15.46:55000) 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3b0b10640 1 -- 172.21.15.46:0/519849219 learned_addr learned my addr 172.21.15.46:0/519849219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3abfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc3ac0b4350 0x7fc3ac0b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:60048/0 (socket says 172.21.15.46:60048) 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3b0b10640 1 -- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 msgr2=0x7fc3ac0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3b0b10640 1 --2- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3b0b10640 1 -- 172.21.15.46:0/519849219 >> v1:172.21.15.78:6789/0 conn(0x7fc3ac0b4350 legacy=0x7fc3ac0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3b0b10640 1 -- 172.21.15.46:0/519849219 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ac0af920 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.044+0000 7fc3ab7fe640 1 -- 172.21.15.46:0/519849219 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2442839892 0 0) 0x7fc3ac0b10d0 con 0x7fc3ac0b4350 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b0b10640 1 --2- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0b0b90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009870 tx=0x7fc39c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edb60ad1de2c5cac server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3ab7fe640 1 -- 172.21.15.46:0/519849219 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39c02f840 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3ab7fe640 1 -- 172.21.15.46:0/519849219 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc39c02f9e0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3ab7fe640 1 -- 172.21.15.46:0/519849219 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39c02fdb0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 -- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac15ecb0 msgr2=0x7fc3ac0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0b0b90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009870 tx=0x7fc39c02ba20 comp rx=0 tx=0).stop 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 -- 172.21.15.46:0/519849219 shutdown_connections 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/519849219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0b0b90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 -- 172.21.15.46:0/519849219 >> 172.21.15.46:0/519849219 conn(0x7fc3ac0b6870 msgr2=0x7fc3ac0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 -- 172.21.15.46:0/519849219 shutdown_connections 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.045+0000 7fc3b2d74640 1 -- 172.21.15.46:0/519849219 wait complete. 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 Processor -- start 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 -- start start 2024-04-15T15:17:10.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac161d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac0b4350 0x7fc3ac147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ac0b47c0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ac0afee0 con 0x7fc3ac0b4350 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.047+0000 7fc3b2d74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ac160cf0 con 0x7fc3ac0b1d30 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0cb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59612/0 (socket says 172.21.15.46:59612) 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 -- 172.21.15.46:0/2226633196 learned_addr learned my addr 172.21.15.46:0/2226633196 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b0b10640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3abfff640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac0b4350 0x7fc3ac147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 -- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 msgr2=0x7fc3ac161d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac161d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 -- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac0b4350 msgr2=0x7fc3ac147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac0b4350 0x7fc3ac147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 -- 172.21.15.46:0/2226633196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ac1483b0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3abfff640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac0b4350 0x7fc3ac147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b1311640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0cb0e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400ec10 tx=0x7fc3a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4018020 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a400cd20 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ac160f40 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40146c0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.048+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ac161500 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.049+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ac068250 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.050+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3a4005d10 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.050+0000 7fc3a97fa640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc39806c920 0x7fc39806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.050+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc3a408c6a0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.050+0000 7fc3b0b10640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc39806c920 0x7fc39806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.051+0000 7fc3b0b10640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc39806c920 0x7fc39806ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc39c036000 tx=0x7fc39c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.054+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3ac068250 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.310+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc3ac0689e0 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.312+0000 7fc3a97fa640 1 -- 172.21.15.46:0/2226633196 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fc3a4022070 con 0x7fc3ac15ecb0 2024-04-15T15:17:10.314 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-04-15T15:17:10.314 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQBiRB1mChVZIBAAvN0/KdZG4Oam2AdTqRjJtw== 2024-04-15T15:17:10.314 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2024-04-15T15:17:10.314 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2024-04-15T15:17:10.315 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-15T15:17:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2024-04-15T15:17:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc39806c920 msgr2=0x7fc39806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc39806c920 0x7fc39806ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc39c036000 tx=0x7fc39c008030 comp rx=0 tx=0).stop 2024-04-15T15:17:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 msgr2=0x7fc3ac0cb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0cb0e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400ec10 tx=0x7fc3a400c490 comp rx=0 tx=0).stop 2024-04-15T15:17:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 shutdown_connections 2024-04-15T15:17:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc39806c920 0x7fc39806ede0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3ac15ecb0 0x7fc3ac0cb0e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ac0b4350 0x7fc3ac147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 --2- 172.21.15.46:0/2226633196 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3ac0b1d30 0x7fc3ac161d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 >> 172.21.15.46:0/2226633196 conn(0x7fc3ac0b6870 msgr2=0x7fc3ac0b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 shutdown_connections 2024-04-15T15:17:10.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.315+0000 7fc3b2d74640 1 -- 172.21.15.46:0/2226633196 wait complete. 2024-04-15T15:17:10.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.431+0000 7f048971c640 1 Processor -- start 2024-04-15T15:17:10.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.431+0000 7f048971c640 1 -- start start 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.432+0000 7f048971c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f048409a9d0 0x7f048409adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.432+0000 7f048971c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04840ced60 con 0x7f048409b780 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.432+0000 7f048971c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f048409b2f0 con 0x7f048415ebb0 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.432+0000 7f048971c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f04840cd8c0 con 0x7f048409a9d0 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.432+0000 7f04837fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f048409b780 0x7f04840b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48820/0 (socket says 172.21.15.46:48820) 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.432+0000 7f04837fe640 1 -- 172.21.15.46:0/1281065393 learned_addr learned my addr 172.21.15.46:0/1281065393 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50822358 0 0) 0x7f0474001490 con 0x7f048409b780 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0470003520 con 0x7f048409b780 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0482ffd640 1 --2- 172.21.15.46:0/1281065393 >> v2:172.21.15.78:3300/0 conn(0x7f048409a9d0 0x7f048409adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1201705906 0 0) 0x7f0470003520 con 0x7f048409b780 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 >> v2:172.21.15.78:3300/0 conn(0x7f048409a9d0 msgr2=0x7f048409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 --2- 172.21.15.46:0/1281065393 >> v2:172.21.15.78:3300/0 conn(0x7f048409a9d0 0x7f048409adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 >> v1:172.21.15.78:6789/0 conn(0x7f048415ebb0 legacy=0x7f048406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04840cccb0 con 0x7f048409b780 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f0474001bc0 con 0x7f048409b780 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 >> v1:172.21.15.46:6789/0 conn(0x7f048409b780 legacy=0x7f04840b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700039b0 0x7f0470003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0470004800 0x7f0470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04700051c0 0x7f047000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04840cd8c0 con 0x7f04700051c0 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f047000c060 con 0x7f0470004800 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f047000c260 con 0x7f04700039b0 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04837fe640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04700051c0 0x7f047000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04827fc640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0470004800 0x7f0470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f0482ffd640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700039b0 0x7f0470003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04837fe640 1 -- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700039b0 msgr2=0x7f0470003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04837fe640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700039b0 0x7f0470003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04837fe640 1 -- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0470004800 msgr2=0x7f0470004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04837fe640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0470004800 0x7f0470004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.433+0000 7f04837fe640 1 -- 172.21.15.46:0/1281065393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0470003520 con 0x7f04700051c0 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.434+0000 7f04837fe640 1 -- 172.21.15.46:0/1281065393 reap_dead start 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.434+0000 7f04837fe640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04700051c0 0x7f047000b9c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f047400aa20 tx=0x7f0470005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55606ac08480bb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.434+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474038070 con 0x7f04700051c0 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.434+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04740072f0 con 0x7f04700051c0 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.434+0000 7f0481ffb640 1 -- 172.21.15.46:0/1281065393 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047400a670 con 0x7f04700051c0 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 -- 172.21.15.46:0/1281065393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04700051c0 msgr2=0x7f047000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04700051c0 0x7f047000b9c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f047400aa20 tx=0x7f0470005830 comp rx=0 tx=0).stop 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 -- 172.21.15.46:0/1281065393 shutdown_connections 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 --2- 172.21.15.46:0/1281065393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04700051c0 0x7f047000b9c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 -- 172.21.15.46:0/1281065393 >> 172.21.15.46:0/1281065393 conn(0x7f04840b3fb0 msgr2=0x7f04840b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 -- 172.21.15.46:0/1281065393 shutdown_connections 2024-04-15T15:17:10.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.435+0000 7f048971c640 1 -- 172.21.15.46:0/1281065393 wait complete. 2024-04-15T15:17:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.436+0000 7f048971c640 1 Processor -- start 2024-04-15T15:17:10.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.436+0000 7f048971c640 1 -- start start 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f048971c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700051c0 0x7f04840cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f048971c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04841f5470 0x7f0484070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f048971c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 0x7f0484071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f048971c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04840cd550 con 0x7f04841f58f0 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f048971c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f04840cd270 con 0x7f04841f5470 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f048971c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f04840b5310 con 0x7f04700051c0 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 0x7f0484071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 0x7f0484071040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59638/0 (socket says 172.21.15.46:59638) 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 -- 172.21.15.46:0/3214868774 learned_addr learned my addr 172.21.15.46:0/3214868774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700051c0 0x7f04840cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 -- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700051c0 msgr2=0x7f04840cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700051c0 0x7f04840cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 -- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04841f5470 msgr2=0x7f0484070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04827fc640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04841f5470 0x7f0484070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04841f5470 0x7f0484070b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04837fe640 1 -- 172.21.15.46:0/3214868774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04840c6e40 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700051c0 0x7f04840cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.437+0000 7f04827fc640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04841f5470 0x7f0484070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.438+0000 7f04837fe640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 0x7f0484071040 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0474036040 tx=0x7f0474001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.438+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474038040 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.438+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0474042070 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.438+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04840c7120 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.438+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474001490 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.438+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0484160520 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.439+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0484160720 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.440+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04740072f0 con 0x7f04841f58f0 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.440+0000 7f047b7fe640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f046406c8f0 0x7f046406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.440+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f04740ad790 con 0x7f04841f58f0 2024-04-15T15:17:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.440+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f046406c8f0 0x7f046406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.441+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f046406c8f0 0x7f046406edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f047c009a40 tx=0x7f047c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.445+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0484160720 con 0x7f04841f58f0 2024-04-15T15:17:10.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.668+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04840690f0 con 0x7f04841f58f0 2024-04-15T15:17:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.669+0000 7f047b7fe640 1 -- 172.21.15.46:0/3214868774 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5175 (secure 0 0 0) 0x7f04840ced60 con 0x7f04841f58f0 2024-04-15T15:17:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:10.671 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:03.437143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5454":{"gid":5454,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/964535021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":964535021},{"type":"v1","addr":"172.21.15.46:6835","nonce":964535021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5454,"qdb_cluster":[5454]},"id":2}]} 2024-04-15T15:17:10.671 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f046406c8f0 msgr2=0x7f046406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f046406c8f0 0x7f046406edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f047c009a40 tx=0x7f047c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 msgr2=0x7f0484071040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 0x7f0484071040 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f0474036040 tx=0x7f0474001360 comp rx=0 tx=0).stop 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 shutdown_connections 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f046406c8f0 0x7f046406edb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04841f58f0 0x7f0484071040 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04841f5470 0x7f0484070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 --2- 172.21.15.46:0/3214868774 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f04700051c0 0x7f04840cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 >> 172.21.15.46:0/3214868774 conn(0x7f04840b3fb0 msgr2=0x7f048415ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 shutdown_connections 2024-04-15T15:17:10.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:10.671+0000 7f048971c640 1 -- 172.21.15.46:0/3214868774 wait complete. 2024-04-15T15:17:11.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:17:00.418468+0000', 'modified': '2024-04-15T15:17:03.437143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5454}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5454': {'gid': 5454, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/964535021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 964535021}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 964535021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5454, 'qdb_cluster': [5454]} 2024-04-15T15:17:11.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:17:11.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-15T15:17:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.194+0000 7fbed3f22640 1 Processor -- start 2024-04-15T15:17:11.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.195+0000 7fbed3f22640 1 -- start start 2024-04-15T15:17:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.195+0000 7fbed3f22640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbecc15e790 0x7fbecc0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.195+0000 7fbed3f22640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 0x7fbecc0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.195+0000 7fbed3f22640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbecc15eb70 con 0x7fbecc0b7be0 2024-04-15T15:17:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.195+0000 7fbed3f22640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fbecc0d3790 con 0x7fbecc15e790 2024-04-15T15:17:11.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.195+0000 7fbed3f22640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc0d45b0 con 0x7fbecc0d1cf0 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed1cbe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbecc15e790 0x7fbecc0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 0x7fbecc0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed14bd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbecc0b7be0 0x7fbecc0d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:60070/0 (socket says 172.21.15.46:60070) 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed1cbe640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fbecc15e790 0x7fbecc0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:55060/0 (socket says 172.21.15.46:55060) 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed14bd640 1 -- 172.21.15.46:0/1519143467 learned_addr learned my addr 172.21.15.46:0/1519143467 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 0x7fbecc0d4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:55868/0 (socket says 172.21.15.46:55868) 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed0cbc640 1 -- 172.21.15.46:0/1519143467 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092762853 0 0) 0x7fbecc15eb70 con 0x7fbecc0b7be0 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed0cbc640 1 -- 172.21.15.46:0/1519143467 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb4003520 con 0x7fbecc0b7be0 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 msgr2=0x7fbecc0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 0x7fbecc0d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1519143467 >> v1:172.21.15.78:6789/0 conn(0x7fbecc0b7be0 legacy=0x7fbecc0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1519143467 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc0d3c80 con 0x7fbecc15e790 2024-04-15T15:17:11.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.196+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 0x7fbecc0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.197+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc15e790 0x7fbecc0b76a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0d3910 tx=0x7fbebc02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e14a7524b97b4347 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.197+0000 7fbed0cbc640 1 -- 172.21.15.46:0/1519143467 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc039070 con 0x7fbecc15e790 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.197+0000 7fbed0cbc640 1 -- 172.21.15.46:0/1519143467 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc02cb20 con 0x7fbecc15e790 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.197+0000 7fbed0cbc640 1 -- 172.21.15.46:0/1519143467 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc034440 con 0x7fbecc15e790 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 -- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc15e790 msgr2=0x7fbecc0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc15e790 0x7fbecc0b76a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0d3910 tx=0x7fbebc02bd70 comp rx=0 tx=0).stop 2024-04-15T15:17:11.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 -- 172.21.15.46:0/1519143467 shutdown_connections 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc0d1cf0 0x7fbecc0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1519143467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc15e790 0x7fbecc0b76a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 -- 172.21.15.46:0/1519143467 >> 172.21.15.46:0/1519143467 conn(0x7fbecc067b40 msgr2=0x7fbecc067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 -- 172.21.15.46:0/1519143467 shutdown_connections 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.198+0000 7fbed3f22640 1 -- 172.21.15.46:0/1519143467 wait complete. 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.199+0000 7fbed3f22640 1 Processor -- start 2024-04-15T15:17:11.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.199+0000 7fbed3f22640 1 -- start start 2024-04-15T15:17:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed3f22640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 0x7fbecc0caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed3f22640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc0d1cf0 0x7fbecc14f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed3f22640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc15e790 0x7fbecc0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed3f22640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc0d4f80 con 0x7fbecc0b7be0 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed3f22640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc0d4240 con 0x7fbecc0d1cf0 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed3f22640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc0d5100 con 0x7fbecc15e790 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 0x7fbecc0caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 0x7fbecc0caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59652/0 (socket says 172.21.15.46:59652) 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1141078930 learned_addr learned my addr 172.21.15.46:0/1141078930 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed14bd640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc0d1cf0 0x7fbecc14f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc15e790 0x7fbecc0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc15e790 msgr2=0x7fbecc0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc15e790 0x7fbecc0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc0d1cf0 msgr2=0x7fbecc14f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc0d1cf0 0x7fbecc14f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1141078930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc0c9a30 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc15e790 0x7fbecc0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.200+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 0x7fbecc0caad0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbebc037040 tx=0x7fbebc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.201+0000 7fbeca7fc640 1 -- 172.21.15.46:0/1141078930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc039050 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.201+0000 7fbeca7fc640 1 -- 172.21.15.46:0/1141078930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc045070 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.201+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc0c9d10 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.201+0000 7fbeca7fc640 1 -- 172.21.15.46:0/1141078930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc009450 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.201+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc0ca2d0 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.202+0000 7fbeca7fc640 1 -- 172.21.15.46:0/1141078930 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbebc03e850 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.203+0000 7fbeca7fc640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbeb806c8f0 0x7fbeb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.203+0000 7fbeca7fc640 1 -- 172.21.15.46:0/1141078930 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbebc0ae990 con 0x7fbecc0b7be0 2024-04-15T15:17:11.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.203+0000 7fbed14bd640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbeb806c8f0 0x7fbeb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.203+0000 7fbed14bd640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbeb806c8f0 0x7fbeb806edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0cb430 tx=0x7fbec0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 Processor -- start 2024-04-15T15:17:11.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 -- start start 2024-04-15T15:17:11.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 0x5587d5db3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5de3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5db42f0 0x5587d5de40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587d5d1e9c0 con 0x5587d5de0e10 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5587d5cc5650 con 0x5587d5db42f0 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.204+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x5587d5d1a980 con 0x5587d5de11f0 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 0x5587d5db3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 0x5587d5db3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59658/0 (socket says 172.21.15.46:59658) 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1107967215 learned_addr learned my addr 172.21.15.46:0/1107967215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed14bd640 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5de3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5db42f0 0x5587d5de40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 msgr2=0x5587d5de3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5de3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5db42f0 msgr2=0x5587d5de40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5db42f0 0x5587d5de40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 -- 172.21.15.46:0/1107967215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587d5a312d0 con 0x5587d5de0e10 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5db42f0 0x5587d5de40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 0x5587d5db3db0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0acc70 tx=0x7fbebc02cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde76d3a71cde61b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbeb37fe640 1 -- 172.21.15.46:0/1107967215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc07cc70 con 0x5587d5de0e10 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbeb37fe640 1 -- 172.21.15.46:0/1107967215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc03a040 con 0x5587d5de0e10 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.205+0000 7fbeb37fe640 1 -- 172.21.15.46:0/1107967215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc07c570 con 0x5587d5de0e10 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1107967215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 msgr2=0x5587d5db3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 0x5587d5db3db0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbebc0acc70 tx=0x7fbebc02cb20 comp rx=0 tx=0).stop 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1107967215 shutdown_connections 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5db42f0 0x5587d5de40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5de3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1107967215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de0e10 0x5587d5db3db0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1107967215 >> 172.21.15.46:0/1107967215 conn(0x5587d5db6020 msgr2=0x5587d5dfeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1107967215 shutdown_connections 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1107967215 wait complete. 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.206+0000 7fbed6bdf740 1 -- start start 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 Processor -- start 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de0e10 0x5587d5dff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5ddd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587d5cc5650 con 0x5587d5de84d0 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x5587d5cd8a10 con 0x5587d5de0e10 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.207+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x5587d5d1e9c0 con 0x5587d5de11f0 2024-04-15T15:17:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed6bdf740 10 client.0 _ll_register_callbacks cb 0x5587d5d14250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59660/0 (socket says 172.21.15.46:59660) 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 -- 172.21.15.46:0/2605478488 learned_addr learned my addr 172.21.15.46:0/2605478488 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de0e10 0x5587d5dff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed14bd640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5ddd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 -- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 msgr2=0x5587d5ddd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5ddd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 -- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de0e10 msgr2=0x5587d5dff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de0e10 0x5587d5dff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 -- 172.21.15.46:0/2605478488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587d5afb6b0 con 0x5587d5de84d0 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed24bf640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de8950 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009d50 tx=0x7fbec40136b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5746ac8464ee2e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbeb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/2605478488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec401a030 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/2605478488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbec4016670 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbebc03a040 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/2605478488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fbec4026050 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed6bdf740 10 client.5640 fetch_fsmap learned FSMap version 17 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.208+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5587d5de5700 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/2605478488 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc03a040 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/2605478488 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7fbec4029020 con 0x5587d5de84d0 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed6bdf740 10 client.5640 fetch_fsmap finished waiting for FSMap version 17 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed6bdf740 10 client.5640 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed6bdf740 20 client.5640 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5de6aa0 0x5587d5e1d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed6bdf740 4 client.5640 mds_command: new command op to 5454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 --> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587d5d2c3e0 con 0x5587d5de6aa0 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5de6aa0 0x5587d5e1d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbeb0ff9640 10 client.5640 ms_handle_connect on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.209+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5de6aa0 0x5587d5e1d710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca8bcda816a9cad0 server_cookie=2521546749d3c428 in_seq=0 out_seq=0 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/2605478488 <== mds.0 v2:172.21.15.46:6833/964535021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbebc09be50 con 0x5587d5de6aa0 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbeb0ff9640 10 client.5640 handle_command_reply: tid=0 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 client.5640 shutdown 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 msgr2=0x5587d5de8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de8950 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009d50 tx=0x7fbec40136b0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 shutdown_connections 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5de6aa0 0x5587d5e1d710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de8950 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de11f0 0x5587d5ddd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbeb0ff9640 0 client.5640 ms_handle_reset on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2605478488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de0e10 0x5587d5dff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 >> 172.21.15.46:0/2605478488 conn(0x5587d5db42f0 msgr2=0x5587d5db8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 shutdown_connections 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2605478488 wait complete. 2024-04-15T15:17:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.210+0000 7fbed6bdf740 20 client.5640 trim_cache size 0 max 16384 2024-04-15T15:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.228+0000 7fbed6bdf740 1 Processor -- start 2024-04-15T15:17:11.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 -- start start 2024-04-15T15:17:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5db91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5e0e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc201340 con 0x5587d5de84d0 2024-04-15T15:17:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc200380 con 0x5587d5de11f0 2024-04-15T15:17:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc0c9440 con 0x5587d5de0e10 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5e0e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5e0e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59662/0 (socket says 172.21.15.46:59662) 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.229+0000 7fbed24bf640 1 -- 172.21.15.46:0/2904741020 learned_addr learned my addr 172.21.15.46:0/2904741020 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5db91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed14bd640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed24bf640 1 -- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 msgr2=0x5587d5db91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed24bf640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5db91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed24bf640 1 -- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 msgr2=0x5587d5de3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed24bf640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed24bf640 1 -- 172.21.15.46:0/2904741020 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587d5db8880 con 0x5587d5de84d0 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5db91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed14bd640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:11.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed24bf640 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5e0e110 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbec4013e80 tx=0x7fbec4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7455337f3b537c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbeb2ffd640 1 -- 172.21.15.46:0/2904741020 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec402f070 con 0x5587d5de84d0 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbeb2ffd640 1 -- 172.21.15.46:0/2904741020 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec4016910 con 0x5587d5de84d0 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbeb2ffd640 1 -- 172.21.15.46:0/2904741020 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec401a040 con 0x5587d5de84d0 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2904741020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 msgr2=0x5587d5e0e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5e0e110 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fbec4013e80 tx=0x7fbec4011000 comp rx=0 tx=0).stop 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2904741020 shutdown_connections 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5e0e110 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/2904741020 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5db91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.230+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2904741020 >> 172.21.15.46:0/2904741020 conn(0x5587d5db42f0 msgr2=0x5587d5dbadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.231+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2904741020 shutdown_connections 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.231+0000 7fbed6bdf740 1 -- 172.21.15.46:0/2904741020 wait complete. 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.231+0000 7fbed6bdf740 1 -- start start 2024-04-15T15:17:11.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.231+0000 7fbed6bdf740 1 Processor -- start 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5ddfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5587d5de7260 con 0x5587d5de84d0 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1ffd60 con 0x5587d5de11f0 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x5587d5ddcd30 con 0x5587d5de0e10 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed6bdf740 10 client.0 _ll_register_callbacks cb 0x7fbecc06e880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59664/0 (socket says 172.21.15.46:59664) 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 -- 172.21.15.46:0/1356561184 learned_addr learned my addr 172.21.15.46:0/1356561184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed14bd640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5ddfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 -- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 msgr2=0x5587d5ddfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5ddfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 -- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 msgr2=0x5587d5de3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed24bf640 1 -- 172.21.15.46:0/1356561184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587d5ddf720 con 0x5587d5de84d0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed14bd640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.232+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5ddfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed24bf640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de42c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbec4013e80 tx=0x7fbec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce18d4ab48049ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbeb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/1356561184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4038060 con 0x5587d5de84d0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/1356561184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbec4033420 con 0x5587d5de84d0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec4016910 con 0x5587d5de84d0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/1356561184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fbec403e050 con 0x5587d5de84d0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 10 client.5646 fetch_fsmap learned FSMap version 17 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x5587d5e797f0 con 0x5587d5de84d0 2024-04-15T15:17:11.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/1356561184 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec4016910 con 0x5587d5de84d0 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/1356561184 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7fbec4041020 con 0x5587d5de84d0 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 10 client.5646 fetch_fsmap finished waiting for FSMap version 17 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 10 client.5646 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 20 client.5646 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5e10d30 0x5587d5e11110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 4 client.5646 mds_command: new command op to 5454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:17:11.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 --> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] -- command(tid 0: {"prefix": "status"}) -- 0x5587d5d26250 con 0x5587d5e10d30 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.233+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5e10d30 0x5587d5e11110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed1cbe640 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5e10d30 0x5587d5e11110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90b0e04c55ea5ba server_cookie=505cb33e2a85e2f3 in_seq=0 out_seq=0 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbeb0ff9640 10 client.5646 ms_handle_connect on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbeb0ff9640 1 -- 172.21.15.46:0/1356561184 <== mds.0 v2:172.21.15.46:6833/964535021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5587d5d26250 con 0x5587d5e10d30 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbeb0ff9640 10 client.5646 handle_command_reply: tid=0 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 client.5646 shutdown 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 msgr2=0x5587d5de42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de42c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbec4013e80 tx=0x7fbec40046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 shutdown_connections 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x5587d5e10d30 0x5587d5e11110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5587d5de84d0 0x5587d5de42c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587d5de11f0 0x5587d5de3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbeb0ff9640 0 client.5646 ms_handle_reset on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:11.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 --2- 172.21.15.46:0/1356561184 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x5587d5de0e10 0x5587d5ddfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 >> 172.21.15.46:0/1356561184 conn(0x5587d5db42f0 msgr2=0x5587d5dbadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 shutdown_connections 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 1 -- 172.21.15.46:0/1356561184 wait complete. 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.234+0000 7fbed6bdf740 20 client.5646 trim_cache size 0 max 16384 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "95f1e89e-6559-430a-8622-bd018dcc0afd", 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5454, 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-15T15:17:11.244 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 10.179282754999999, 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 17, 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 39, 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 39, 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 17.239305957999999 2024-04-15T15:17:11.245 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbeb806c8f0 msgr2=0x7fbeb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbeb806c8f0 0x7fbeb806edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0cb430 tx=0x7fbec0009040 comp rx=0 tx=0).stop 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 msgr2=0x7fbecc0caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 0x7fbecc0caad0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbebc037040 tx=0x7fbebc004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 shutdown_connections 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbeb806c8f0 0x7fbeb806edb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbecc15e790 0x7fbecc0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbecc0d1cf0 0x7fbecc14f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 --2- 172.21.15.46:0/1141078930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbecc0b7be0 0x7fbecc0caad0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 >> 172.21.15.46:0/1141078930 conn(0x7fbecc067b40 msgr2=0x7fbecc0b80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 shutdown_connections 2024-04-15T15:17:11.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.235+0000 7fbed3f22640 1 -- 172.21.15.46:0/1141078930 wait complete. 2024-04-15T15:17:11.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:17:11.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.351+0000 7fbc2f952640 1 Processor -- start 2024-04-15T15:17:11.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.351+0000 7fbc2f952640 1 -- start start 2024-04-15T15:17:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2f952640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2806caf0 0x7fbc280d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2f952640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2806cff0 con 0x7fbc2809b0e0 2024-04-15T15:17:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2f952640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2806d2d0 con 0x7fbc280b1590 2024-04-15T15:17:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2f952640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbc2806dc00 con 0x7fbc2806caf0 2024-04-15T15:17:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2ceed640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbc2809b0e0 0x7fbc2815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48828/0 (socket says 172.21.15.46:48828) 2024-04-15T15:17:11.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2ceed640 1 -- 172.21.15.46:0/2686426873 learned_addr learned my addr 172.21.15.46:0/2686426873 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2deef640 1 --2- 172.21.15.46:0/2686426873 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2806caf0 0x7fbc280d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224628018 0 0) 0x7fbc2806cff0 con 0x7fbc2809b0e0 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc10003520 con 0x7fbc2809b0e0 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258892300 0 0) 0x7fbc2806d2d0 con 0x7fbc280b1590 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc2806cff0 con 0x7fbc280b1590 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2deef640 1 -- 172.21.15.46:0/2686426873 >> v1:172.21.15.78:6789/0 conn(0x7fbc280b1590 legacy=0x7fbc2809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2deef640 1 -- 172.21.15.46:0/2686426873 >> v1:172.21.15.46:6789/0 conn(0x7fbc2809b0e0 legacy=0x7fbc2815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.352+0000 7fbc2deef640 1 -- 172.21.15.46:0/2686426873 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2806d5b0 con 0x7fbc2806caf0 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.353+0000 7fbc2deef640 1 --2- 172.21.15.46:0/2686426873 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2806caf0 0x7fbc280d16b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc180097d0 tx=0x7fbc1802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba5fcdc7e601a02d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.353+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18035020 con 0x7fbc2806caf0 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.353+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc18031890 con 0x7fbc2806caf0 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.353+0000 7fbc27fff640 1 -- 172.21.15.46:0/2686426873 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18031c40 con 0x7fbc2806caf0 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 -- 172.21.15.46:0/2686426873 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2806caf0 msgr2=0x7fbc280d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 --2- 172.21.15.46:0/2686426873 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2806caf0 0x7fbc280d16b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbc180097d0 tx=0x7fbc1802b930 comp rx=0 tx=0).stop 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 -- 172.21.15.46:0/2686426873 shutdown_connections 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 --2- 172.21.15.46:0/2686426873 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2806caf0 0x7fbc280d16b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 -- 172.21.15.46:0/2686426873 >> 172.21.15.46:0/2686426873 conn(0x7fbc280b3e90 msgr2=0x7fbc280b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 -- 172.21.15.46:0/2686426873 shutdown_connections 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.354+0000 7fbc2f952640 1 -- 172.21.15.46:0/2686426873 wait complete. 2024-04-15T15:17:11.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.355+0000 7fbc2f952640 1 Processor -- start 2024-04-15T15:17:11.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.355+0000 7fbc2f952640 1 -- start start 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2f952640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc2806caf0 0x7fbc281dcf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2f952640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2809b0e0 0x7fbc280b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2f952640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 0x7fbc2806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2f952640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc280d0b20 con 0x7fbc280b1590 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2f952640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbc280d09a0 con 0x7fbc2806caf0 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2f952640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbc280b1970 con 0x7fbc2809b0e0 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 0x7fbc2806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 0x7fbc2806fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59680/0 (socket says 172.21.15.46:59680) 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 -- 172.21.15.46:0/1941624411 learned_addr learned my addr 172.21.15.46:0/1941624411 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2d6ee640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc2806caf0 0x7fbc281dcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2ceed640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2809b0e0 0x7fbc280b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 -- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2809b0e0 msgr2=0x7fbc280b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2809b0e0 0x7fbc280b79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 -- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc2806caf0 msgr2=0x7fbc281dcf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc2806caf0 0x7fbc281dcf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 -- 172.21.15.46:0/1941624411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc281e0ce0 con 0x7fbc280b1590 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2d6ee640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc2806caf0 0x7fbc281dcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc2deef640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 0x7fbc2806fdf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbc1802b970 tx=0x7fbc180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.356+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18035040 con 0x7fbc280b1590 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.357+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc18030d20 con 0x7fbc280b1590 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.357+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc281e0fc0 con 0x7fbc280b1590 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.357+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18006e10 con 0x7fbc280b1590 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.357+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc281e1580 con 0x7fbc280b1590 2024-04-15T15:17:11.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.358+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc280689e0 con 0x7fbc280b1590 2024-04-15T15:17:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.358+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc18038020 con 0x7fbc280b1590 2024-04-15T15:17:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.359+0000 7fbc25ffb640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1406c8f0 0x7fbc1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.359+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbc180ae280 con 0x7fbc280b1590 2024-04-15T15:17:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.359+0000 7fbc2d6ee640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1406c8f0 0x7fbc1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.359+0000 7fbc2d6ee640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1406c8f0 0x7fbc1406edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbc20001610 tx=0x7fbc20009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.363+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc1802d040 con 0x7fbc280b1590 2024-04-15T15:17:11.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.587+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbc280b4a80 con 0x7fbc280b1590 2024-04-15T15:17:11.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.588+0000 7fbc25ffb640 1 -- 172.21.15.46:0/1941624411 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fbc1807bf60 con 0x7fbc280b1590 2024-04-15T15:17:11.590 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1406c8f0 msgr2=0x7fbc1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1406c8f0 0x7fbc1406edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbc20001610 tx=0x7fbc20009040 comp rx=0 tx=0).stop 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 msgr2=0x7fbc2806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 0x7fbc2806fdf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbc1802b970 tx=0x7fbc180046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 shutdown_connections 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbc1406c8f0 0x7fbc1406edb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc280b1590 0x7fbc2806fdf0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbc2809b0e0 0x7fbc280b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 --2- 172.21.15.46:0/1941624411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc2806caf0 0x7fbc281dcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 >> 172.21.15.46:0/1941624411 conn(0x7fbc280b3e90 msgr2=0x7fbc281e5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 shutdown_connections 2024-04-15T15:17:11.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11.590+0000 7fbc2f952640 1 -- 172.21.15.46:0/1941624411 wait complete. 2024-04-15T15:17:11.602 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:17:11.602 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:17:11.602 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-15T15:17:11.603 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:17:11.603 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:11.603 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:17:11.603 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:17:11.603 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:17:11.603 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:17:11.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:11.603 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:11.619 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42621sec preferred_lft 42621sec 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:11.620 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 12:e9:35:22:9f:8f brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::50d2:bff:fe30:917d/64 scope link 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 1e:a1:9f:32:6c:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::1ca1:9fff:fe32:6cb0/64 scope link 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 12:e9:35:22:9f:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::10e9:35ff:fe22:9f8f/64 scope link 2024-04-15T15:17:11.621 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:11.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:11.622 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:17:11.686 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-15T15:17:11.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:11.688 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-15T15:17:11.745 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-15T15:17:11.745 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:17:11.745 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:17:11.745 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:17:11.745 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:17:11.746 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:17:11.746 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:17:11.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:17:11.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:17:11.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:17:11.746 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:17:11.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:17:11.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:11.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:11.884 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:17:11.902 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:17:11.902 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:17:11.903 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:17:11.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:11.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-04-15T15:17:11.972 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:17:11.972 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:11.972 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:12.027 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:17:12.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:12.027 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:12.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:17:12.152 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:12.208 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:17:12.209 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:17:12.210 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:12.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.211 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:17:12.264 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:17:12.264 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:17:12.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:17:12.334 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:17:12.391 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:17:12.392 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:12.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.393 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:17:12.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-04-15T15:17:12.393+0000 7f617ee68480 -1 init, newargv = 0x563ec2a98de0 newargc=15 2024-04-15T15:17:12.395 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54036]: starting ceph client 2024-04-15T15:17:12.399 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54036]: starting fuse 2024-04-15T15:17:12.410 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:17:12.411 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:17:12.411 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:17:12.412 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-04-15T15:17:12.413 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-04-15T15:17:12.413 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-04-15T15:17:12.413 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:17:12.413 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:12.413 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-04-15T15:17:12.413 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.54036.asok 2024-04-15T15:17:12.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:12.543 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54036.asok 2024-04-15T15:17:12.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54036.asok status 2024-04-15T15:17:12.684 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:12.684 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-04-15T15:17:12.684 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:17:12.684 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:17:12.684 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-04-15T15:17:12.684 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "54036", 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5658, 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5658 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:17:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3966970179 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3966970179 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5658 192.168.0.1:0/3966970179", 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/3966970179", 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-04-15T15:17:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 17, 2024-04-15T15:17:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 39, 2024-04-15T15:17:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:17:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-04-15T15:17:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-04-15T15:17:12.687 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:12.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.693 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-04-15T15:17:12.693 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:12.709 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-04-15T15:17:12.710 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:12.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.710 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:12.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.921+0000 7fe9f9174640 1 Processor -- start 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.922+0000 7fe9f9174640 1 -- start start 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.922+0000 7fe9f9174640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe9f4072d80 0x7fe9f4165aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.922+0000 7fe9f9174640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9f40b4200 con 0x7fe9f4155330 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.922+0000 7fe9f9174640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9f40b44e0 con 0x7fe9f415bc30 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.922+0000 7fe9f9174640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe9f40b52a0 con 0x7fe9f4072d80 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.922+0000 7fe9f2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe9f4155330 0x7fe9f40b39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48834/0 (socket says 172.21.15.46:48834) 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/2807919958 learned_addr learned my addr 172.21.15.46:0/2807919958 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f359e640 1 --2- 172.21.15.46:0/2807919958 >> v2:172.21.15.78:3300/0 conn(0x7fe9f4072d80 0x7fe9f4165aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437316434 0 0) 0x7fe9f40b4200 con 0x7fe9f4155330 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9e0003520 con 0x7fe9f4155330 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2443012233 0 0) 0x7fe9e0003520 con 0x7fe9f4155330 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 >> v2:172.21.15.78:3300/0 conn(0x7fe9f4072d80 msgr2=0x7fe9f4165aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 --2- 172.21.15.46:0/2807919958 >> v2:172.21.15.78:3300/0 conn(0x7fe9f4072d80 0x7fe9f4165aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 >> v1:172.21.15.78:6789/0 conn(0x7fe9f415bc30 legacy=0x7fe9f4072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f40b47c0 con 0x7fe9f4155330 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fe9e40019f0 con 0x7fe9f4155330 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 >> v1:172.21.15.46:6789/0 conn(0x7fe9f4155330 legacy=0x7fe9f40b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9e00051c0 0x7fe9e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f40b52a0 con 0x7fe9e00039b0 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e000c060 con 0x7fe9e0004800 2024-04-15T15:17:12.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e000c260 con 0x7fe9e00051c0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f259c640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.923+0000 7fe9f359e640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9e00051c0 0x7fe9e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9e00051c0 msgr2=0x7fe9e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9e00051c0 0x7fe9e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e0004800 msgr2=0x7fe9e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/2807919958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e0003520 con 0x7fe9e00039b0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f359e640 1 -- 172.21.15.46:0/2807919958 reap_dead start 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400a860 tx=0x7fe9e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac8ffff791eea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e400a480 con 0x7fe9e00039b0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e4034630 con 0x7fe9e00039b0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.924+0000 7fe9f1d9b640 1 -- 172.21.15.46:0/2807919958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4034aa0 con 0x7fe9e00039b0 2024-04-15T15:17:12.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 -- 172.21.15.46:0/2807919958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 msgr2=0x7fe9e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400a860 tx=0x7fe9e0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 -- 172.21.15.46:0/2807919958 shutdown_connections 2024-04-15T15:17:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 --2- 172.21.15.46:0/2807919958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 -- 172.21.15.46:0/2807919958 >> 172.21.15.46:0/2807919958 conn(0x7fe9f40b1f60 msgr2=0x7fe9f40b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 -- 172.21.15.46:0/2807919958 shutdown_connections 2024-04-15T15:17:12.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.925+0000 7fe9f9174640 1 -- 172.21.15.46:0/2807919958 wait complete. 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.926+0000 7fe9f9174640 1 Processor -- start 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 -- start start 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9f40ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9f4151890 0x7fe9f4157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9f4151d10 0x7fe9f41582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f40b4d80 con 0x7fe9e00039b0 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f415c0c0 con 0x7fe9f4151890 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f9174640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f40b39f0 con 0x7fe9f4151d10 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9f40ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9f40ce5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59688/0 (socket says 172.21.15.46:59688) 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/943632024 learned_addr learned my addr 172.21.15.46:0/943632024 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:12.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f359e640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9f4151d10 0x7fe9f41582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.927+0000 7fe9f259c640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9f4151890 0x7fe9f4157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9f4151d10 msgr2=0x7fe9f41582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9f4151d10 0x7fe9f41582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9f4151890 msgr2=0x7fe9f4157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9f4151890 0x7fe9f4157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f2d9d640 1 -- 172.21.15.46:0/943632024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f40ceb30 con 0x7fe9e00039b0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f259c640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9f4151890 0x7fe9f4157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f2d9d640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9f40ce5f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400a860 tx=0x7fe9f415b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4041020 con 0x7fe9e00039b0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e4034430 con 0x7fe9e00039b0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f40cd550 con 0x7fe9e00039b0 2024-04-15T15:17:12.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4004d60 con 0x7fe9e00039b0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.928+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f40cdb10 con 0x7fe9e00039b0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.929+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f40689e0 con 0x7fe9e00039b0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.930+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe9e4044020 con 0x7fe9e00039b0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.930+0000 7fe9eb7fe640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe9d406c8f0 0x7fe9d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.930+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe9e40ada10 con 0x7fe9e00039b0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.930+0000 7fe9f259c640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe9d406c8f0 0x7fe9d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.931+0000 7fe9f259c640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe9d406c8f0 0x7fe9d406edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc000c00 tx=0x7fe9dc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:12.934+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9e404b030 con 0x7fe9e00039b0 2024-04-15T15:17:13.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.162+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe9f40afcb0 con 0x7fe9e00039b0 2024-04-15T15:17:13.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.162+0000 7fe9eb7fe640 1 -- 172.21.15.46:0/943632024 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7fe9e407bd40 con 0x7fe9e00039b0 2024-04-15T15:17:13.164 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe9d406c8f0 msgr2=0x7fe9d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe9d406c8f0 0x7fe9d406edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc000c00 tx=0x7fe9dc009040 comp rx=0 tx=0).stop 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 msgr2=0x7fe9f40ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9f40ce5f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe9e400a860 tx=0x7fe9f415b120 comp rx=0 tx=0).stop 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 shutdown_connections 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe9d406c8f0 0x7fe9d406edb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe9f4151d10 0x7fe9f41582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9f4151890 0x7fe9f4157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 --2- 172.21.15.46:0/943632024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9e00039b0 0x7fe9f40ce5f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 >> 172.21.15.46:0/943632024 conn(0x7fe9f40b1f60 msgr2=0x7fe9f406f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 shutdown_connections 2024-04-15T15:17:13.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:13.164+0000 7fe9f9174640 1 -- 172.21.15.46:0/943632024 wait complete. 2024-04-15T15:17:13.176 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:17:13.176 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:17:13.176 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-04-15T15:17:13.177 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:17:13.177 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:13.177 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:17:13.177 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:17:13.177 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:17:13.177 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:17:13.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.177 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-04-15T15:17:13.199 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:17:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:17:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:17:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:17:13.200 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.200 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:13.201 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:13.201 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-04-15T15:17:13.201 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:13.201 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:13.201 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-04-15T15:17:13.201 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42605sec preferred_lft 42605sec 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-04-15T15:17:13.202 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 2e:a8:38:23:6a:89 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::60de:b7ff:fe5c:109c/64 scope link 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.203 INFO:teuthology.orchestra.run.smithi078.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:13.204 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 9e:e5:17:fe:02:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:13.204 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::9ce5:17ff:fefe:2aa/64 scope link 2024-04-15T15:17:13.204 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.204 INFO:teuthology.orchestra.run.smithi078.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:13.204 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 2e:a8:38:23:6a:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:13.204 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::2ca8:38ff:fe23:6a89/64 scope link 2024-04-15T15:17:13.205 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:13.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.205 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:17:13.272 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-04-15T15:17:13.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.274 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-04-15T15:17:13.336 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-04-15T15:17:13.336 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:17:13.336 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:17:13.336 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:17:13.336 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:17:13.337 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:17:13.337 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:17:13.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:17:13.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:17:13.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:17:13.337 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:17:13.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:17:13.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:13.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.487 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:17:13.503 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:17:13.503 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:17:13.503 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:17:13.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-04-15T15:17:13.573 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:17:13.574 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:13.574 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:13.630 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:17:13.631 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:13.631 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:13.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:17:13.754 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:13.811 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:17:13.811 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:17:13.811 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:17:13.811 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:17:13.811 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:13.812 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:13.813 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:13.814 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:13.814 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:13.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.814 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:17:13.868 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:17:13.868 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:17:13.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:17:13.939 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:17:13.996 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:13.997 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:13.999 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:13.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:13.999 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:17:14.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:2024-04-15T15:17:13.997+0000 7f7f9c985480 -1 init, newargv = 0x55a634c8dcd0 newargc=15 2024-04-15T15:17:14.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[44755]: starting ceph client 2024-04-15T15:17:14.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[44755]: starting fuse 2024-04-15T15:17:14.015 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:17:14.015 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:17:14.016 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-04-15T15:17:14.017 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:17:14.018 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:14.018 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.1")) 2024-04-15T15:17:14.018 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-04-15T15:17:14.138 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.1.44755.asok 2024-04-15T15:17:14.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:14.148 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44755.asok 2024-04-15T15:17:14.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:14.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44755.asok status 2024-04-15T15:17:14.286 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "1", 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "44755", 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-04-15T15:17:14.287 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5449, 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5449 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2895781748 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:17:14.288 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2895781748 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.5449 192.168.0.1:0/2895781748", 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/2895781748", 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 39, 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-04-15T15:17:14.289 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-04-15T15:17:14.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:14.296 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2024-04-15T15:17:14.296 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:14.312 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2024-04-15T15:17:14.312 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:14.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:14.312 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:14.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 25 2024-04-15T15:17:14.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.499+0000 7f72d5a0d640 1 Processor -- start 2024-04-15T15:17:14.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.499+0000 7f72d5a0d640 1 -- start start 2024-04-15T15:17:14.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72d5a0d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f72d015e790 0x7f72d00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72d5a0d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72d00b7be0 0x7f72d00d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72d5a0d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72d5a0d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f72d00d1f90 con 0x7f72d00b7be0 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72d5a0d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f72d00d3790 con 0x7f72d015e790 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72d5a0d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f72d00d45b0 con 0x7f72d00d1b20 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72d00b7be0 0x7f72d00d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f72d00b7be0 0x7f72d00d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59704/0 (socket says 172.21.15.46:59704) 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 -- 172.21.15.46:0/751271274 learned_addr learned my addr 172.21.15.46:0/751271274 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72cf7fe640 1 --2- 172.21.15.46:0/751271274 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ceffd640 1 --2- 172.21.15.46:0/751271274 >> v2:172.21.15.78:3300/0 conn(0x7f72d015e790 0x7f72d00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 -- 172.21.15.46:0/751271274 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 msgr2=0x7f72d00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 --2- 172.21.15.46:0/751271274 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 -- 172.21.15.46:0/751271274 >> v2:172.21.15.78:3300/0 conn(0x7f72d015e790 msgr2=0x7f72d00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 --2- 172.21.15.46:0/751271274 >> v2:172.21.15.78:3300/0 conn(0x7f72d015e790 0x7f72d00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.500+0000 7f72ce7fc640 1 -- 172.21.15.46:0/751271274 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d015eb70 con 0x7f72d00b7be0 2024-04-15T15:17:14.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.501+0000 7f72ce7fc640 1 --2- 172.21.15.46:0/751271274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00d15e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f72d00d4730 tx=0x7f72b802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a20fdd18f6d482a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.501+0000 7f72cdffb640 1 -- 172.21.15.46:0/751271274 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8038070 con 0x7f72d00b7be0 2024-04-15T15:17:14.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.501+0000 7f72cdffb640 1 -- 172.21.15.46:0/751271274 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b8031740 con 0x7f72d00b7be0 2024-04-15T15:17:14.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.501+0000 7f72cdffb640 1 -- 172.21.15.46:0/751271274 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b802f050 con 0x7f72d00b7be0 2024-04-15T15:17:14.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 -- 172.21.15.46:0/751271274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 msgr2=0x7f72d00d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/751271274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00d15e0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f72d00d4730 tx=0x7f72b802b7e0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 -- 172.21.15.46:0/751271274 shutdown_connections 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/751271274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00d15e0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/751271274 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/751271274 >> v2:172.21.15.78:3300/0 conn(0x7f72d015e790 0x7f72d00b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 -- 172.21.15.46:0/751271274 >> 172.21.15.46:0/751271274 conn(0x7f72d0067b40 msgr2=0x7f72d0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 -- 172.21.15.46:0/751271274 shutdown_connections 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.502+0000 7f72d5a0d640 1 -- 172.21.15.46:0/751271274 wait complete. 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.503+0000 7f72d5a0d640 1 Processor -- start 2024-04-15T15:17:14.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.503+0000 7f72d5a0d640 1 -- start start 2024-04-15T15:17:14.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72d5a0d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72d5a0d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72d5a0d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72d015e790 0x7f72d015cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72d5a0d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72d00d4240 con 0x7f72d00b7be0 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72d5a0d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f72d00d3f60 con 0x7f72d015e790 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72d5a0d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f72d00d4f80 con 0x7f72d00d1b20 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59712/0 (socket says 172.21.15.46:59712) 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 -- 172.21.15.46:0/3249166718 learned_addr learned my addr 172.21.15.46:0/3249166718 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ce7fc640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 -- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 msgr2=0x7f72d00ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 -- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72d015e790 msgr2=0x7f72d015cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72d015e790 0x7f72d015cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.504+0000 7f72ceffd640 1 -- 172.21.15.46:0/3249166718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d00ab970 con 0x7f72d00b7be0 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.505+0000 7f72ceffd640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00cdb20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f72c800d5d0 tx=0x7f72c800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.505+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80125f0 con 0x7f72d00b7be0 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.505+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c8019070 con 0x7f72d00b7be0 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.505+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d00abc50 con 0x7f72d00b7be0 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.505+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80129f0 con 0x7f72d00b7be0 2024-04-15T15:17:14.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.505+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d00ac210 con 0x7f72d00b7be0 2024-04-15T15:17:14.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.506+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d00ac650 con 0x7f72d00b7be0 2024-04-15T15:17:14.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.506+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f72c8012b90 con 0x7f72d00b7be0 2024-04-15T15:17:14.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.507+0000 7f72c7fff640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72b006c8f0 0x7f72b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.507+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f72c808c160 con 0x7f72d00b7be0 2024-04-15T15:17:14.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.507+0000 7f72ce7fc640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72b006c8f0 0x7f72b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.507+0000 7f72ce7fc640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72b006c8f0 0x7f72b006edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f72b8035000 tx=0x7f72b8008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.511+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72c805a360 con 0x7f72d00b7be0 2024-04-15T15:17:14.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.695+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f72d00689e0 con 0x7f72d00b7be0 2024-04-15T15:17:14.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.698+0000 7f72c7fff640 1 -- 172.21.15.46:0/3249166718 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7f72c8059c60 con 0x7f72d00b7be0 2024-04-15T15:17:14.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72b006c8f0 msgr2=0x7f72b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72b006c8f0 0x7f72b006edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f72b8035000 tx=0x7f72b8008030 comp rx=0 tx=0).stop 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 msgr2=0x7f72d00cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00cdb20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f72c800d5d0 tx=0x7f72c800da90 comp rx=0 tx=0).stop 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 shutdown_connections 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f72b006c8f0 0x7f72b006edb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72d015e790 0x7f72d015cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f72d00d1b20 0x7f72d00ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 --2- 172.21.15.46:0/3249166718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d00b7be0 0x7f72d00cdb20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 >> 172.21.15.46:0/3249166718 conn(0x7f72d0067b40 msgr2=0x7f72d00b80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 shutdown_connections 2024-04-15T15:17:14.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.701+0000 7f72d5a0d640 1 -- 172.21.15.46:0/3249166718 wait complete. 2024-04-15T15:17:14.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_cache_quiesce_splitauth true 2024-04-15T15:17:14.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.819+0000 7fe56b3f2640 1 Processor -- start 2024-04-15T15:17:14.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.819+0000 7fe56b3f2640 1 -- start start 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56b3f2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe564070940 0x7fe56409b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56b3f2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe564160310 0x7fe564070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56b3f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe5640d0da0 0x7fe5640d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56b3f2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe5640d05e0 con 0x7fe564070940 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56b3f2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe564071260 con 0x7fe564160310 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56b3f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe56415f720 con 0x7fe5640d0da0 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe564070940 0x7fe56409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe564070940 0x7fe56409b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59724/0 (socket says 172.21.15.46:59724) 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 -- 172.21.15.46:0/3071809687 learned_addr learned my addr 172.21.15.46:0/3071809687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56998f640 1 --2- 172.21.15.46:0/3071809687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe5640d0da0 0x7fe5640d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56898d640 1 --2- 172.21.15.46:0/3071809687 >> v2:172.21.15.78:3300/0 conn(0x7fe564160310 0x7fe564070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 -- 172.21.15.46:0/3071809687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe5640d0da0 msgr2=0x7fe5640d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 --2- 172.21.15.46:0/3071809687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe5640d0da0 0x7fe5640d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 -- 172.21.15.46:0/3071809687 >> v2:172.21.15.78:3300/0 conn(0x7fe564160310 msgr2=0x7fe564070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 --2- 172.21.15.46:0/3071809687 >> v2:172.21.15.78:3300/0 conn(0x7fe564160310 0x7fe564070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56918e640 1 -- 172.21.15.46:0/3071809687 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56415fa00 con 0x7fe564070940 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.820+0000 7fe56898d640 1 --2- 172.21.15.46:0/3071809687 >> v2:172.21.15.78:3300/0 conn(0x7fe564160310 0x7fe564070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.821+0000 7fe56918e640 1 --2- 172.21.15.46:0/3071809687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe564070940 0x7fe56409b880 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe5640734f0 tx=0x7fe56002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5293b5fff4d5f74d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.821+0000 7fe55bfff640 1 -- 172.21.15.46:0/3071809687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56002ca70 con 0x7fe564070940 2024-04-15T15:17:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.821+0000 7fe55bfff640 1 -- 172.21.15.46:0/3071809687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56002cc10 con 0x7fe564070940 2024-04-15T15:17:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.821+0000 7fe55bfff640 1 -- 172.21.15.46:0/3071809687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5600343f0 con 0x7fe564070940 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 -- 172.21.15.46:0/3071809687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe564070940 msgr2=0x7fe56409b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/3071809687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe564070940 0x7fe56409b880 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fe5640734f0 tx=0x7fe56002bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 -- 172.21.15.46:0/3071809687 shutdown_connections 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/3071809687 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe5640d0da0 0x7fe5640d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/3071809687 >> v2:172.21.15.78:3300/0 conn(0x7fe564160310 0x7fe564070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/3071809687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe564070940 0x7fe56409b880 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 -- 172.21.15.46:0/3071809687 >> 172.21.15.46:0/3071809687 conn(0x7fe5640b3f40 msgr2=0x7fe5640b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 -- 172.21.15.46:0/3071809687 shutdown_connections 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.822+0000 7fe56b3f2640 1 -- 172.21.15.46:0/3071809687 wait complete. 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.823+0000 7fe56b3f2640 1 Processor -- start 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.823+0000 7fe56b3f2640 1 -- start start 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56b3f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe564070940 0x7fe564157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56b3f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 0x7fe5640b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56b3f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe564160310 0x7fe5640c6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56b3f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe564160770 con 0x7fe5640d0da0 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56b3f2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640d0880 con 0x7fe564160310 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56b3f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640d1750 con 0x7fe564070940 2024-04-15T15:17:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 0x7fe5640b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 0x7fe5640b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59734/0 (socket says 172.21.15.46:59734) 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 -- 172.21.15.46:0/1293486961 learned_addr learned my addr 172.21.15.46:0/1293486961 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56998f640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe564160310 0x7fe5640c6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56918e640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe564070940 0x7fe564157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 -- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe564070940 msgr2=0x7fe564157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe564070940 0x7fe564157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 -- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe564160310 msgr2=0x7fe5640c6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe564160310 0x7fe5640c6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 -- 172.21.15.46:0/1293486961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5640c7030 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.824+0000 7fe56898d640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 0x7fe5640b0090 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe55400d7d0 tx=0x7fe55400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.825+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554004030 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.825+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5540041d0 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.825+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5640c7300 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.825+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5540096a0 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.825+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5640c78c0 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.826+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5640689e0 con 0x7fe5640d0da0 2024-04-15T15:17:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.826+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe554009840 con 0x7fe5640d0da0 2024-04-15T15:17:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.827+0000 7fe559ffb640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe54806c8f0 0x7fe54806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.827+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe55408be30 con 0x7fe5640d0da0 2024-04-15T15:17:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.827+0000 7fe56918e640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe54806c8f0 0x7fe54806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.828+0000 7fe56918e640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe54806c8f0 0x7fe54806edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe560035000 tx=0x7fe560030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:14.831+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe554059a80 con 0x7fe5640d0da0 2024-04-15T15:17:15.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.017+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mds_cache_quiesce_splitauth}] v 0) -- 0x7fe564070720 con 0x7fe5640d0da0 2024-04-15T15:17:15.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.020+0000 7fe559ffb640 1 -- 172.21.15.46:0/1293486961 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_cache_quiesce_splitauth}]=0 v12)=0 v12) ==== 128+0+0 (secure 0 0 0) 0x7fe55405dd80 con 0x7fe5640d0da0 2024-04-15T15:17:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe54806c8f0 msgr2=0x7fe54806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe54806c8f0 0x7fe54806edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe560035000 tx=0x7fe560030000 comp rx=0 tx=0).stop 2024-04-15T15:17:15.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 msgr2=0x7fe5640b0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 0x7fe5640b0090 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe55400d7d0 tx=0x7fe55400dc90 comp rx=0 tx=0).stop 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 shutdown_connections 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe54806c8f0 0x7fe54806edb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe564160310 0x7fe5640c6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5640d0da0 0x7fe5640b0090 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 --2- 172.21.15.46:0/1293486961 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe564070940 0x7fe564157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 >> 172.21.15.46:0/1293486961 conn(0x7fe5640b3f40 msgr2=0x7fe5640b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 shutdown_connections 2024-04-15T15:17:15.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.022+0000 7fe56b3f2640 1 -- 172.21.15.46:0/1293486961 wait complete. 2024-04-15T15:17:15.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol_quiesce --mode=777 2024-04-15T15:17:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.142+0000 7f089d7ba640 1 Processor -- start 2024-04-15T15:17:15.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.142+0000 7f089d7ba640 1 -- start start 2024-04-15T15:17:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f089d7ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f089815c430 0x7f089806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f089d7ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08980d0470 0x7f08980d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f089d7ba640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08980cbdb0 con 0x7f08980cb910 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f089d7ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f08980b4e80 con 0x7f08980d0470 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f089d7ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f08980d0df0 con 0x7f089815c430 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08980d0470 0x7f08980d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08980d0470 0x7f08980d08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59742/0 (socket says 172.21.15.46:59742) 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 -- 172.21.15.46:0/3769067070 learned_addr learned my addr 172.21.15.46:0/3769067070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08967fc640 1 --1- 172.21.15.46:0/3769067070 >> v1:172.21.15.46:6789/0 conn(0x7f08980cb910 0x7f08980b4130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48846/0 (socket says 172.21.15.46:48846) 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f0896ffd640 1 --2- 172.21.15.46:0/3769067070 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f089815c430 0x7f089806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 -- 172.21.15.46:0/3769067070 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f089815c430 msgr2=0x7f089806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 --2- 172.21.15.46:0/3769067070 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f089815c430 0x7f089806dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 -- 172.21.15.46:0/3769067070 >> v1:172.21.15.46:6789/0 conn(0x7f08980cb910 legacy=0x7f08980b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f08977fe640 1 -- 172.21.15.46:0/3769067070 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08980cc090 con 0x7f08980d0470 2024-04-15T15:17:15.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.143+0000 7f0895ffb640 1 -- 172.21.15.46:0/3769067070 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570535345 0 0) 0x7f08980cbdb0 con 0x7f08980cb910 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.144+0000 7f08977fe640 1 --2- 172.21.15.46:0/3769067070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980d0470 0x7f08980d08b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0888009870 tx=0x7f088802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d696c390adab2785 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.144+0000 7f0895ffb640 1 -- 172.21.15.46:0/3769067070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888035020 con 0x7f08980d0470 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.144+0000 7f0895ffb640 1 -- 172.21.15.46:0/3769067070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0888031640 con 0x7f08980d0470 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.144+0000 7f0895ffb640 1 -- 172.21.15.46:0/3769067070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088802d040 con 0x7f08980d0470 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 -- 172.21.15.46:0/3769067070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980d0470 msgr2=0x7f08980d08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 --2- 172.21.15.46:0/3769067070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980d0470 0x7f08980d08b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0888009870 tx=0x7f088802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 -- 172.21.15.46:0/3769067070 shutdown_connections 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 --2- 172.21.15.46:0/3769067070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980d0470 0x7f08980d08b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 --2- 172.21.15.46:0/3769067070 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f089815c430 0x7f089806dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 -- 172.21.15.46:0/3769067070 >> 172.21.15.46:0/3769067070 conn(0x7f089806bbc0 msgr2=0x7f089806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 -- 172.21.15.46:0/3769067070 shutdown_connections 2024-04-15T15:17:15.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.145+0000 7f089d7ba640 1 -- 172.21.15.46:0/3769067070 wait complete. 2024-04-15T15:17:15.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.146+0000 7f089d7ba640 1 Processor -- start 2024-04-15T15:17:15.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.146+0000 7f089d7ba640 1 -- start start 2024-04-15T15:17:15.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f089d7ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08980cb910 0x7f08981ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f089d7ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08980d0470 0x7f08980b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f089d7ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 0x7f08980b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f089d7ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f08980cc930 con 0x7f089815c430 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f089d7ba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08980cc650 con 0x7f08980d0470 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f089d7ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f089815c810 con 0x7f08980cb910 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 0x7f08980b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 0x7f08980b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59754/0 (socket says 172.21.15.46:59754) 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 -- 172.21.15.46:0/1577919420 learned_addr learned my addr 172.21.15.46:0/1577919420 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:15.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f0896ffd640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08980cb910 0x7f08981ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08967fc640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08980d0470 0x7f08980b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 -- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08980cb910 msgr2=0x7f08981ee250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08980cb910 0x7f08981ee250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 -- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08980d0470 msgr2=0x7f08980b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08980d0470 0x7f08980b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.147+0000 7f08977fe640 1 -- 172.21.15.46:0/1577919420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08981f4410 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.148+0000 7f08977fe640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 0x7f08980b2380 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0888009870 tx=0x7f08880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.148+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888040070 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.148+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08880399f0 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.148+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08981f46f0 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.148+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888035040 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.148+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08981f4cb0 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.149+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08980689e0 con 0x7f089815c430 2024-04-15T15:17:15.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.149+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f088804b060 con 0x7f089815c430 2024-04-15T15:17:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.150+0000 7f088f7fe640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f087806c8f0 0x7f087806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.150+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f08880ad440 con 0x7f089815c430 2024-04-15T15:17:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.150+0000 7f0896ffd640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f087806c8f0 0x7f087806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.151+0000 7f0896ffd640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f087806c8f0 0x7f087806edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08980ca300 tx=0x7f08900098d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.154+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f088807b090 con 0x7f089815c430 2024-04-15T15:17:15.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.366+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f0898073e60 con 0x7f087806c8f0 2024-04-15T15:17:15.373 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:17:15.372+0000 7f4990fbe640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:17:15.373 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:17:15.372+0000 7f4990fbe640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:17:15.374 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:17:15.372+0000 7f4990fbe640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:17:15.374 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:17:15.372+0000 7f4990fbe640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:17:15.374 INFO:tasks.ceph.mgr.x.smithi078.stderr:2024-04-15T15:17:15.372+0000 7f4990fbe640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:17:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.402+0000 7f088f7fe640 1 -- 172.21.15.46:0/1577919420 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0898073e60 con 0x7f087806c8f0 2024-04-15T15:17:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f087806c8f0 msgr2=0x7f087806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f087806c8f0 0x7f087806edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08980ca300 tx=0x7f08900098d0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 msgr2=0x7f08980b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 0x7f08980b2380 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0888009870 tx=0x7f08880046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 shutdown_connections 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f087806c8f0 0x7f087806edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f089815c430 0x7f08980b2380 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08980d0470 0x7f08980b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 --2- 172.21.15.46:0/1577919420 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08980cb910 0x7f08981ee250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 >> 172.21.15.46:0/1577919420 conn(0x7f089806bbc0 msgr2=0x7f0898074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 shutdown_connections 2024-04-15T15:17:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.404+0000 7f089d7ba640 1 -- 172.21.15.46:0/1577919420 wait complete. 2024-04-15T15:17:15.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_quiesce 2024-04-15T15:17:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.522+0000 7fae60fe2640 1 Processor -- start 2024-04-15T15:17:15.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.522+0000 7fae60fe2640 1 -- start start 2024-04-15T15:17:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.523+0000 7fae60fe2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fae5c15e700 0x7fae5c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.523+0000 7fae60fe2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae5c0d2360 0x7fae5c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.523+0000 7fae60fe2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15eae0 0x7fae5c0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.523+0000 7fae60fe2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fae5c0b53b0 con 0x7fae5c0d2360 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.523+0000 7fae60fe2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fae5c0b47a0 con 0x7fae5c15e700 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.523+0000 7fae60fe2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c0b4a80 con 0x7fae5c15eae0 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae5c0d2360 0x7fae5c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae5c0d2360 0x7fae5c0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59766/0 (socket says 172.21.15.46:59766) 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 -- 172.21.15.46:0/3465902285 learned_addr learned my addr 172.21.15.46:0/3465902285 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae5ad9d640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15eae0 0x7fae5c0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae5a59c640 1 --2- 172.21.15.46:0/3465902285 >> v2:172.21.15.78:3300/0 conn(0x7fae5c15e700 0x7fae5c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 -- 172.21.15.46:0/3465902285 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15eae0 msgr2=0x7fae5c0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15eae0 0x7fae5c0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 -- 172.21.15.46:0/3465902285 >> v2:172.21.15.78:3300/0 conn(0x7fae5c15e700 msgr2=0x7fae5c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 --2- 172.21.15.46:0/3465902285 >> v2:172.21.15.78:3300/0 conn(0x7fae5c15e700 0x7fae5c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae59d9b640 1 -- 172.21.15.46:0/3465902285 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c0b4d60 con 0x7fae5c0d2360 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae5ad9d640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15eae0 0x7fae5c0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.524+0000 7fae5a59c640 1 --2- 172.21.15.46:0/3465902285 >> v2:172.21.15.78:3300/0 conn(0x7fae5c15e700 0x7fae5c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae59d9b640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c0d2b60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fae5c06c6c0 tx=0x7fae5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a30fb1b22a90b90 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae5959a640 1 -- 172.21.15.46:0/3465902285 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50035020 con 0x7fae5c0d2360 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae5959a640 1 -- 172.21.15.46:0/3465902285 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae50031870 con 0x7fae5c0d2360 2024-04-15T15:17:15.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae5959a640 1 -- 172.21.15.46:0/3465902285 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50031c40 con 0x7fae5c0d2360 2024-04-15T15:17:15.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 -- 172.21.15.46:0/3465902285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 msgr2=0x7fae5c0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c0d2b60 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fae5c06c6c0 tx=0x7fae5002b910 comp rx=0 tx=0).stop 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 -- 172.21.15.46:0/3465902285 shutdown_connections 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c0d2b60 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 --2- 172.21.15.46:0/3465902285 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15eae0 0x7fae5c0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 --2- 172.21.15.46:0/3465902285 >> v2:172.21.15.78:3300/0 conn(0x7fae5c15e700 0x7fae5c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.525+0000 7fae60fe2640 1 -- 172.21.15.46:0/3465902285 >> 172.21.15.46:0/3465902285 conn(0x7fae5c08bf60 msgr2=0x7fae5c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.526+0000 7fae60fe2640 1 -- 172.21.15.46:0/3465902285 shutdown_connections 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.526+0000 7fae60fe2640 1 -- 172.21.15.46:0/3465902285 wait complete. 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.527+0000 7fae60fe2640 1 Processor -- start 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.527+0000 7fae60fe2640 1 -- start start 2024-04-15T15:17:15.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.527+0000 7fae60fe2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15e700 0x7fae5c0b9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae5c15eae0 0x7fae5c0d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c0b5d80 con 0x7fae5c0d2360 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c0b4450 con 0x7fae5c15eae0 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c0d1d40 con 0x7fae5c15e700 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c1517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59778/0 (socket says 172.21.15.46:59778) 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 -- 172.21.15.46:0/2668712796 learned_addr learned my addr 172.21.15.46:0/2668712796 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae59d9b640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15e700 0x7fae5c0b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5ad9d640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae5c15eae0 0x7fae5c0d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 -- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15e700 msgr2=0x7fae5c0b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15e700 0x7fae5c0b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 -- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae5c15eae0 msgr2=0x7fae5c0d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae5c15eae0 0x7fae5c0d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 -- 172.21.15.46:0/2668712796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c0cdd00 con 0x7fae5c0d2360 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae59d9b640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15e700 0x7fae5c0b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae5a59c640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c1517c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c7c0 tx=0x7fae4c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c005b90 con 0x7fae5c0d2360 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4c005d30 con 0x7fae5c0d2360 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c006250 con 0x7fae5c0d2360 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5c0cdfe0 con 0x7fae5c0d2360 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.528+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5c0ce5a0 con 0x7fae5c0d2360 2024-04-15T15:17:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.529+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae5c0ce8b0 con 0x7fae5c0d2360 2024-04-15T15:17:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.530+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae4c0063f0 con 0x7fae5c0d2360 2024-04-15T15:17:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.530+0000 7fae4affd640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae3c06c8f0 0x7fae3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.530+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fae4c08bde0 con 0x7fae5c0d2360 2024-04-15T15:17:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.530+0000 7fae59d9b640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae3c06c8f0 0x7fae3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.531+0000 7fae59d9b640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae3c06c8f0 0x7fae3c06edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fae50036000 tx=0x7fae50008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.534+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae4c015020 con 0x7fae5c0d2360 2024-04-15T15:17:15.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.748+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "target": ["mon-mgr", ""]}) -- 0x7fae5c0689e0 con 0x7fae3c06c8f0 2024-04-15T15:17:15.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.754+0000 7fae4affd640 1 -- 172.21.15.46:0/2668712796 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+70 (secure 0 0 0) 0x7fae5c0689e0 con 0x7fae3c06c8f0 2024-04-15T15:17:15.756 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda 2024-04-15T15:17:15.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae3c06c8f0 msgr2=0x7fae3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae3c06c8f0 0x7fae3c06edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fae50036000 tx=0x7fae50008030 comp rx=0 tx=0).stop 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 msgr2=0x7fae5c1517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c1517c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c7c0 tx=0x7fae4c00cc80 comp rx=0 tx=0).stop 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 shutdown_connections 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae3c06c8f0 0x7fae3c06edb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae5c15eae0 0x7fae5c0d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae5c15e700 0x7fae5c0b9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 --2- 172.21.15.46:0/2668712796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae5c0d2360 0x7fae5c1517c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.755+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 >> 172.21.15.46:0/2668712796 conn(0x7fae5c08bf60 msgr2=0x7fae5c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.756+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 shutdown_connections 2024-04-15T15:17:15.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.756+0000 7fae60fe2640 1 -- 172.21.15.46:0/2668712796 wait complete. 2024-04-15T15:17:15.768 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:15.769 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:15.785 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:15.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 Processor -- start 2024-04-15T15:17:15.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 -- start start 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a0c15e790 0x7f8a0c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0c15eb70 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8a0c0d3790 con 0x7f8a0c15e790 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.927+0000 7f8a1109a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c0d45b0 con 0x7f8a0c0d1cf0 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a0c15e790 0x7f8a0c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8a0c15e790 0x7f8a0c0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59784/0 (socket says 172.21.15.46:59784) 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3774018885 learned_addr learned my addr 172.21.15.46:0/3774018885 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:15.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0b59e640 1 --2- 172.21.15.46:0/3774018885 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3774018885 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c0d1cf0 msgr2=0x7f8a0c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 --2- 172.21.15.46:0/3774018885 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3774018885 >> v1:172.21.15.78:6789/0 conn(0x7f8a0c0b7be0 legacy=0x7f8a0c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.928+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3774018885 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c0d3c80 con 0x7f8a0c15e790 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a0ad9d640 1 --2- 172.21.15.46:0/3774018885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c15e790 0x7f8a0c0b76a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0d3910 tx=0x7f89fc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6bfe6b5a6befc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a09d9b640 1 -- 172.21.15.46:0/3774018885 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc038070 con 0x7f8a0c15e790 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a09d9b640 1 -- 172.21.15.46:0/3774018885 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89fc031710 con 0x7f8a0c15e790 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a09d9b640 1 -- 172.21.15.46:0/3774018885 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc031ae0 con 0x7f8a0c15e790 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 -- 172.21.15.46:0/3774018885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c15e790 msgr2=0x7f8a0c0b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3774018885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c15e790 0x7f8a0c0b76a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0d3910 tx=0x7f89fc02b7b0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 -- 172.21.15.46:0/3774018885 shutdown_connections 2024-04-15T15:17:15.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3774018885 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3774018885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c15e790 0x7f8a0c0b76a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 -- 172.21.15.46:0/3774018885 >> 172.21.15.46:0/3774018885 conn(0x7f8a0c067b40 msgr2=0x7f8a0c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 -- 172.21.15.46:0/3774018885 shutdown_connections 2024-04-15T15:17:15.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.929+0000 7f8a1109a640 1 -- 172.21.15.46:0/3774018885 wait complete. 2024-04-15T15:17:15.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.931+0000 7f8a1109a640 1 Processor -- start 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.931+0000 7f8a1109a640 1 -- start start 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 0x7f8a0c1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c15e790 0x7f8a0c0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c0d4f80 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c0d4240 con 0x7f8a0c0d1cf0 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c0d5100 con 0x7f8a0c15e790 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 0x7f8a0c1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 0x7f8a0c1543f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59788/0 (socket says 172.21.15.46:59788) 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3378924878 learned_addr learned my addr 172.21.15.46:0/3378924878 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0b59e640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c15e790 0x7f8a0c0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0a59c640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c15e790 msgr2=0x7f8a0c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c15e790 0x7f8a0c0cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a0c0d1cf0 msgr2=0x7f8a0c0cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 -- 172.21.15.46:0/3378924878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c0c8120 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0a59c640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a0ad9d640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 0x7f8a0c1543f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f89fc02b7f0 tx=0x7f89fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc043070 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89fc03f870 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c0c8400 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.932+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc038040 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.933+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c0c89c0 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.934+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0c0689e0 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.934+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f89fc046070 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.934+0000 7f89f37fe640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f89f806c8f0 0x7f89f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.934+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f89fc0ad500 con 0x7f8a0c0b7be0 2024-04-15T15:17:15.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.935+0000 7f8a0a59c640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f89f806c8f0 0x7f89f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.935+0000 7f8a0a59c640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f89f806c8f0 0x7f89f806edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c1609a0 tx=0x7f8a00009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:15.939+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89fc07b270 con 0x7f8a0c0b7be0 2024-04-15T15:17:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.126+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a0c0cecc0 con 0x7f8a0c0b7be0 2024-04-15T15:17:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.127+0000 7f89f37fe640 1 -- 172.21.15.46:0/3378924878 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7f89fc07f570 con 0x7f8a0c0b7be0 2024-04-15T15:17:16.129 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:16.129 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:02.054907+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f89f806c8f0 msgr2=0x7f89f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f89f806c8f0 0x7f89f806edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c1609a0 tx=0x7f8a00009040 comp rx=0 tx=0).stop 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 msgr2=0x7f8a0c1543f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 0x7f8a0c1543f0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f89fc02b7f0 tx=0x7f89fc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 shutdown_connections 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f89f806c8f0 0x7f89f806edb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8a0c15e790 0x7f8a0c0cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a0c0d1cf0 0x7f8a0c0cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 --2- 172.21.15.46:0/3378924878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a0c0b7be0 0x7f8a0c1543f0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 >> 172.21.15.46:0/3378924878 conn(0x7f8a0c067b40 msgr2=0x7f8a0c0d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 shutdown_connections 2024-04-15T15:17:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.129+0000 7f8a1109a640 1 -- 172.21.15.46:0/3378924878 wait complete. 2024-04-15T15:17:16.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:16.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:16.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.248+0000 7f95a6932640 1 Processor -- start 2024-04-15T15:17:16.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.248+0000 7f95a6932640 1 -- start start 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a6932640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95a006ec00 0x7f95a0160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a6932640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95a015d660 con 0x7f95a0067720 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a6932640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95a015d940 con 0x7f95a015d090 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a6932640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f95a0160130 con 0x7f95a006ec00 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a4ecf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95a006ec00 0x7f95a0160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a4ecf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f95a006ec00 0x7f95a0160b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59804/0 (socket says 172.21.15.46:59804) 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a4ecf640 1 -- 172.21.15.46:0/2867839682 learned_addr learned my addr 172.21.15.46:0/2867839682 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:16.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f959f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f95a0067720 0x7f95a00ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48860/0 (socket says 172.21.15.46:48860) 2024-04-15T15:17:16.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a4ecf640 1 -- 172.21.15.46:0/2867839682 >> v1:172.21.15.78:6789/0 conn(0x7f95a015d090 legacy=0x7f95a006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a4ecf640 1 -- 172.21.15.46:0/2867839682 >> v1:172.21.15.46:6789/0 conn(0x7f95a0067720 legacy=0x7f95a00ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.249+0000 7f95a4ecf640 1 -- 172.21.15.46:0/2867839682 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a0160380 con 0x7f95a006ec00 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.250+0000 7f95a4ecf640 1 --2- 172.21.15.46:0/2867839682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a0160b80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f95a01602b0 tx=0x7f959802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c0164e94f409ef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.250+0000 7f959effd640 1 -- 172.21.15.46:0/2867839682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959802ca70 con 0x7f95a006ec00 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.250+0000 7f959effd640 1 -- 172.21.15.46:0/2867839682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959802cc10 con 0x7f95a006ec00 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.250+0000 7f959effd640 1 -- 172.21.15.46:0/2867839682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980343f0 con 0x7f95a006ec00 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 -- 172.21.15.46:0/2867839682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 msgr2=0x7f95a0160b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 --2- 172.21.15.46:0/2867839682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a0160b80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f95a01602b0 tx=0x7f959802bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 -- 172.21.15.46:0/2867839682 shutdown_connections 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 --2- 172.21.15.46:0/2867839682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a0160b80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 -- 172.21.15.46:0/2867839682 >> 172.21.15.46:0/2867839682 conn(0x7f95a0073360 msgr2=0x7f95a006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 -- 172.21.15.46:0/2867839682 shutdown_connections 2024-04-15T15:17:16.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.251+0000 7f95a6932640 1 -- 172.21.15.46:0/2867839682 wait complete. 2024-04-15T15:17:16.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.252+0000 7f95a6932640 1 Processor -- start 2024-04-15T15:17:16.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.252+0000 7f95a6932640 1 -- start start 2024-04-15T15:17:16.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a6932640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f95a0067720 0x7f95a0151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a6932640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a014f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a6932640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a015d090 0x7f95a015b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a6932640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f95a006f070 con 0x7f95a006ec00 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a6932640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f95a01610c0 con 0x7f95a015d090 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a6932640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f95a00b0460 con 0x7f95a0067720 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a014f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a014f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59812/0 (socket says 172.21.15.46:59812) 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 -- 172.21.15.46:0/561336243 learned_addr learned my addr 172.21.15.46:0/561336243 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959ffff640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f95a0067720 0x7f95a0151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f95a4ecf640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a015d090 0x7f95a015b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 -- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f95a0067720 msgr2=0x7f95a0151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f95a0067720 0x7f95a0151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 -- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a015d090 msgr2=0x7f95a015b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a015d090 0x7f95a015b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.253+0000 7f959f7fe640 1 -- 172.21.15.46:0/561336243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a0158940 con 0x7f95a006ec00 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.254+0000 7f959f7fe640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a014f3e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f959400c400 tx=0x7f959400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.254+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95940074e0 con 0x7f95a006ec00 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.254+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9594007680 con 0x7f95a006ec00 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.254+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594004a00 con 0x7f95a006ec00 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.254+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a01576c0 con 0x7f95a006ec00 2024-04-15T15:17:16.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.254+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a0157c80 con 0x7f95a006ec00 2024-04-15T15:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.255+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a00689e0 con 0x7f95a006ec00 2024-04-15T15:17:16.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.255+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f959401a020 con 0x7f95a006ec00 2024-04-15T15:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.256+0000 7f959cff9640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f958c06c8f0 0x7f958c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.256+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f959408b540 con 0x7f95a006ec00 2024-04-15T15:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.256+0000 7f959ffff640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f958c06c8f0 0x7f958c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.256+0000 7f959ffff640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f958c06c8f0 0x7f958c06edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9590006fd0 tx=0x7f9590008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.260+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f95940597e0 con 0x7f95a006ec00 2024-04-15T15:17:16.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.445+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f95a006e670 con 0x7f95a006ec00 2024-04-15T15:17:16.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.445+0000 7f959cff9640 1 -- 172.21.15.46:0/561336243 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f95940590e0 con 0x7f95a006ec00 2024-04-15T15:17:16.447 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f958c06c8f0 msgr2=0x7f958c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f958c06c8f0 0x7f958c06edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9590006fd0 tx=0x7f9590008040 comp rx=0 tx=0).stop 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 msgr2=0x7f95a014f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a014f3e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f959400c400 tx=0x7f959400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 shutdown_connections 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f958c06c8f0 0x7f958c06edb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a015d090 0x7f95a015b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95a006ec00 0x7f95a014f3e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 --2- 172.21.15.46:0/561336243 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f95a0067720 0x7f95a0151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 >> 172.21.15.46:0/561336243 conn(0x7f95a0073360 msgr2=0x7f95a00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 shutdown_connections 2024-04-15T15:17:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.447+0000 7f95a6932640 1 -- 172.21.15.46:0/561336243 wait complete. 2024-04-15T15:17:16.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:16.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.566+0000 7fab48488640 1 Processor -- start 2024-04-15T15:17:16.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.566+0000 7fab48488640 1 -- start start 2024-04-15T15:17:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab48488640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fab4015f0d0 0x7fab400d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab48488640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab400d05e0 con 0x7fab40070940 2024-04-15T15:17:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab48488640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4015f720 con 0x7fab4009b880 2024-04-15T15:17:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab48488640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fab4009bce0 con 0x7fab4015f0d0 2024-04-15T15:17:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab46224640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fab40070940 0x7fab40160310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48874/0 (socket says 172.21.15.46:48874) 2024-04-15T15:17:16.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab46224640 1 -- 172.21.15.46:0/1202016502 learned_addr learned my addr 172.21.15.46:0/1202016502 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab46a25640 1 --2- 172.21.15.46:0/1202016502 >> v2:172.21.15.78:3300/0 conn(0x7fab4015f0d0 0x7fab400d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011743707 0 0) 0x7fab400d05e0 con 0x7fab40070940 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab2c003520 con 0x7fab40070940 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316785766 0 0) 0x7fab4015f720 con 0x7fab4009b880 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab400d05e0 con 0x7fab4009b880 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1557718611 0 0) 0x7fab2c003520 con 0x7fab40070940 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 >> v2:172.21.15.78:3300/0 conn(0x7fab4015f0d0 msgr2=0x7fab400d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 --2- 172.21.15.46:0/1202016502 >> v2:172.21.15.78:3300/0 conn(0x7fab4015f0d0 0x7fab400d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 >> v1:172.21.15.78:6789/0 conn(0x7fab4009b880 legacy=0x7fab40070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4015fa00 con 0x7fab40070940 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fab3c001670 con 0x7fab40070940 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 >> v1:172.21.15.46:6789/0 conn(0x7fab40070940 legacy=0x7fab40160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c003b40 0x7fab2c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab2c004800 0x7fab2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab2c0051c0 0x7fab2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fab4009bce0 con 0x7fab2c0051c0 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c00c040 con 0x7fab2c003b40 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.567+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fab2c00c240 con 0x7fab2c004800 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab2c0051c0 0x7fab2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab45a23640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab2c004800 0x7fab2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46224640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c003b40 0x7fab2c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 -- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab2c004800 msgr2=0x7fab2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab2c004800 0x7fab2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 -- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c003b40 msgr2=0x7fab2c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c003b40 0x7fab2c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 -- 172.21.15.46:0/1202016502 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c003520 con 0x7fab2c0051c0 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.568+0000 7fab46a25640 1 -- 172.21.15.46:0/1202016502 reap_dead start 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab46a25640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab2c0051c0 0x7fab2c00b9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fab38009fc0 tx=0x7fab2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68955698a67cc51a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38009c10 con 0x7fab2c0051c0 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab3802cc60 con 0x7fab2c0051c0 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab45222640 1 -- 172.21.15.46:0/1202016502 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38003bd0 con 0x7fab2c0051c0 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 -- 172.21.15.46:0/1202016502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab2c0051c0 msgr2=0x7fab2c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab2c0051c0 0x7fab2c00b9c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fab38009fc0 tx=0x7fab2c005830 comp rx=0 tx=0).stop 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 -- 172.21.15.46:0/1202016502 shutdown_connections 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 --2- 172.21.15.46:0/1202016502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab2c0051c0 0x7fab2c00b9c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 -- 172.21.15.46:0/1202016502 >> 172.21.15.46:0/1202016502 conn(0x7fab400b3f40 msgr2=0x7fab400b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 -- 172.21.15.46:0/1202016502 shutdown_connections 2024-04-15T15:17:16.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.569+0000 7fab48488640 1 -- 172.21.15.46:0/1202016502 wait complete. 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 Processor -- start 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 -- start start 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c0051c0 0x7fab400b6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab400afe50 0x7fab400b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 0x7fab4015cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fab40160310 con 0x7fab4015cb40 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fab400d0880 con 0x7fab2c0051c0 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab48488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fab40160490 con 0x7fab400afe50 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab46a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 0x7fab4015cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab46a25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 0x7fab4015cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59834/0 (socket says 172.21.15.46:59834) 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.571+0000 7fab46a25640 1 -- 172.21.15.46:0/2170334674 learned_addr learned my addr 172.21.15.46:0/2170334674 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:16.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46224640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c0051c0 0x7fab400b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab45a23640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab400afe50 0x7fab400b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46a25640 1 -- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab400afe50 msgr2=0x7fab400b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46a25640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab400afe50 0x7fab400b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46a25640 1 -- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c0051c0 msgr2=0x7fab400b6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46a25640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c0051c0 0x7fab400b6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46a25640 1 -- 172.21.15.46:0/2170334674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4015d530 con 0x7fab4015cb40 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab46a25640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 0x7fab4015cff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e200 tx=0x7fab38009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38003990 con 0x7fab4015cb40 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab38044070 con 0x7fab4015cb40 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38003d50 con 0x7fab4015cb40 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab400b6da0 con 0x7fab4015cb40 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.572+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab400b7360 con 0x7fab4015cb40 2024-04-15T15:17:16.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.573+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab400689e0 con 0x7fab4015cb40 2024-04-15T15:17:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.573+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fab38005d10 con 0x7fab4015cb40 2024-04-15T15:17:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.574+0000 7fab36ffd640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fab2406c920 0x7fab2406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.574+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fab380ae330 con 0x7fab4015cb40 2024-04-15T15:17:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.574+0000 7fab46224640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fab2406c920 0x7fab2406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.575+0000 7fab46224640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fab2406c920 0x7fab2406ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fab3c007c20 tx=0x7fab3c009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.578+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab3807c660 con 0x7fab4015cb40 2024-04-15T15:17:16.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.709+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fab380785d0 con 0x7fab4015cb40 2024-04-15T15:17:16.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.763+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fab40070720 con 0x7fab4015cb40 2024-04-15T15:17:16.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.764+0000 7fab36ffd640 1 -- 172.21.15.46:0/2170334674 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7fab380a6c70 con 0x7fab4015cb40 2024-04-15T15:17:16.766 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fab2406c920 msgr2=0x7fab2406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fab2406c920 0x7fab2406ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fab3c007c20 tx=0x7fab3c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 msgr2=0x7fab4015cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 0x7fab4015cff0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fab3800e200 tx=0x7fab38009750 comp rx=0 tx=0).stop 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 shutdown_connections 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fab2406c920 0x7fab2406ede0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab4015cb40 0x7fab4015cff0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fab400afe50 0x7fab400b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 --2- 172.21.15.46:0/2170334674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab2c0051c0 0x7fab400b6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 >> 172.21.15.46:0/2170334674 conn(0x7fab400b3f40 msgr2=0x7fab400b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 shutdown_connections 2024-04-15T15:17:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.766+0000 7fab48488640 1 -- 172.21.15.46:0/2170334674 wait complete. 2024-04-15T15:17:16.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:16.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.884+0000 7fdd00ae4640 1 Processor -- start 2024-04-15T15:17:16.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.884+0000 7fdd00ae4640 1 -- start start 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdd00ae4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdcfc070940 0x7fdcfc09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdd00ae4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 0x7fdcfc070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdd00ae4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcfc0d05e0 con 0x7fdcfc0d0da0 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdd00ae4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fdcfc071260 con 0x7fdcfc070940 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdd00ae4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdcfc0d1940 con 0x7fdcfc160310 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdcfad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdcfc0d0da0 0x7fdcfc0d1230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48876/0 (socket says 172.21.15.46:48876) 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdcfad9d640 1 -- 172.21.15.46:0/2350823523 learned_addr learned my addr 172.21.15.46:0/2350823523 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 0x7fdcfc070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/2350823523 >> v2:172.21.15.78:3300/0 conn(0x7fdcfc070940 0x7fdcfc09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724748762 0 0) 0x7fdcfc0d05e0 con 0x7fdcfc0d0da0 2024-04-15T15:17:16.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.885+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdce8003520 con 0x7fdcfc0d0da0 2024-04-15T15:17:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 907007718 0 0) 0x7fdce8003520 con 0x7fdcfc0d0da0 2024-04-15T15:17:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 msgr2=0x7fdcfc070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 0x7fdcfc070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 >> v2:172.21.15.78:3300/0 conn(0x7fdcfc070940 msgr2=0x7fdcfc09b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 --2- 172.21.15.46:0/2350823523 >> v2:172.21.15.78:3300/0 conn(0x7fdcfc070940 0x7fdcfc09b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcfc15f720 con 0x7fdcfc0d0da0 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fdcec0019f0 con 0x7fdcfc0d0da0 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 >> v1:172.21.15.46:6789/0 conn(0x7fdcfc0d0da0 legacy=0x7fdcfc0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdce80039b0 0x7fdce8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 0x7fdce8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce8004d40 0x7fdce80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc071260 con 0x7fdce80039b0 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc0d1940 con 0x7fdce8004d40 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdce800b9c0 con 0x7fdcfc160310 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdce80039b0 0x7fdce8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 0x7fdce8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce8004d40 0x7fdce80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfa59c640 1 -- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 msgr2=0x7fdce8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc160310 0x7fdce8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfa59c640 1 -- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce8004d40 msgr2=0x7fdce80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce8004d40 0x7fdce80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.886+0000 7fdcfa59c640 1 -- 172.21.15.46:0/2350823523 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce8003520 con 0x7fdce80039b0 2024-04-15T15:17:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdce80039b0 0x7fdce8003dd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009ac0 tx=0x7fdce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a26745c7cc089f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4039070 con 0x7fdce80039b0 2024-04-15T15:17:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf402c710 con 0x7fdce80039b0 2024-04-15T15:17:16.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdcf959a640 1 -- 172.21.15.46:0/2350823523 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4009690 con 0x7fdce80039b0 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdd00ae4640 1 -- 172.21.15.46:0/2350823523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdce80039b0 msgr2=0x7fdce8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/2350823523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdce80039b0 0x7fdce8003dd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009ac0 tx=0x7fdce8005830 comp rx=0 tx=0).stop 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.887+0000 7fdcfad9d640 1 -- 172.21.15.46:0/2350823523 reap_dead start 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.888+0000 7fdd00ae4640 1 -- 172.21.15.46:0/2350823523 shutdown_connections 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.888+0000 7fdd00ae4640 1 -- 172.21.15.46:0/2350823523 >> 172.21.15.46:0/2350823523 conn(0x7fdcfc0b3f40 msgr2=0x7fdcfc0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.888+0000 7fdd00ae4640 1 -- 172.21.15.46:0/2350823523 shutdown_connections 2024-04-15T15:17:16.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.888+0000 7fdd00ae4640 1 -- 172.21.15.46:0/2350823523 wait complete. 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.889+0000 7fdd00ae4640 1 Processor -- start 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.889+0000 7fdd00ae4640 1 -- start start 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce80039b0 0x7fdcfc1f8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc0b6f30 0x7fdcfc14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 0x7fdcfc0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc0d0880 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdcfc15fce0 con 0x7fdce80039b0 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdcfc160770 con 0x7fdcfc0b6f30 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 0x7fdcfc0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 0x7fdcfc0b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59848/0 (socket says 172.21.15.46:59848) 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 -- 172.21.15.46:0/447577611 learned_addr learned my addr 172.21.15.46:0/447577611 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce80039b0 0x7fdcfc1f8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcf9d9b640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc0b6f30 0x7fdcfc14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 -- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc0b6f30 msgr2=0x7fdcfc14c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc0b6f30 0x7fdcfc14c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 -- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce80039b0 msgr2=0x7fdcfc1f8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce80039b0 0x7fdcfc1f8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 -- 172.21.15.46:0/447577611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcfc0b74f0 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcfad9d640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 0x7fdcfc0b3240 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc0d2420 tx=0x7fdcec00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec00ee20 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcec014430 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcfc0b77d0 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcec014820 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.890+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcfc14cd60 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.891+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcfc0b3780 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.892+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fdcec0149c0 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.892+0000 7fdcf2ffd640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdcdc06c9b0 0x7fdcdc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.893+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fdcec08bc90 con 0x7fdcfc0b2dc0 2024-04-15T15:17:16.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.893+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdcdc06c9b0 0x7fdcdc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.893+0000 7fdcfa59c640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdcdc06c9b0 0x7fdcdc06ee70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4036000 tx=0x7fdcf4030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:16.897+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdcec059eb0 con 0x7fdcfc0b2dc0 2024-04-15T15:17:17.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.083+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdcfc0cea30 con 0x7fdcfc0b2dc0 2024-04-15T15:17:17.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.084+0000 7fdcf2ffd640 1 -- 172.21.15.46:0/447577611 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fdcfc0cea30 con 0x7fdcfc0b2dc0 2024-04-15T15:17:17.086 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdcdc06c9b0 msgr2=0x7fdcdc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdcdc06c9b0 0x7fdcdc06ee70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4036000 tx=0x7fdcf4030000 comp rx=0 tx=0).stop 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 msgr2=0x7fdcfc0b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 0x7fdcfc0b3240 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc0d2420 tx=0x7fdcec00e530 comp rx=0 tx=0).stop 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 shutdown_connections 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdcdc06c9b0 0x7fdcdc06ee70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcfc0b2dc0 0x7fdcfc0b3240 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdcfc0b6f30 0x7fdcfc14c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 --2- 172.21.15.46:0/447577611 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdce80039b0 0x7fdcfc1f8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 >> 172.21.15.46:0/447577611 conn(0x7fdcfc0b3f40 msgr2=0x7fdcfc15ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 shutdown_connections 2024-04-15T15:17:17.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.086+0000 7fdd00ae4640 1 -- 172.21.15.46:0/447577611 wait complete. 2024-04-15T15:17:17.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:17.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.203+0000 7fbd47552640 1 Processor -- start 2024-04-15T15:17:17.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.203+0000 7fbd47552640 1 -- start start 2024-04-15T15:17:17.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd47552640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd400d3240 0x7fbd400d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd47552640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd400d1440 0x7fbd400b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd47552640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd400d2480 con 0x7fbd4015e790 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd47552640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbd400d2db0 con 0x7fbd400d1440 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd47552640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbd400d2760 con 0x7fbd400d3240 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd45aef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd400d1440 0x7fbd400b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd45aef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbd400d1440 0x7fbd400b74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59854/0 (socket says 172.21.15.46:59854) 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd45aef640 1 -- 172.21.15.46:0/2246595034 learned_addr learned my addr 172.21.15.46:0/2246595034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.204+0000 7fbd44aed640 1 --2- 172.21.15.46:0/2246595034 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd400d3240 0x7fbd400d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.205+0000 7fbd45aef640 1 -- 172.21.15.46:0/2246595034 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd400d3240 msgr2=0x7fbd400d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.205+0000 7fbd45aef640 1 --2- 172.21.15.46:0/2246595034 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd400d3240 0x7fbd400d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.205+0000 7fbd45aef640 1 -- 172.21.15.46:0/2246595034 >> v1:172.21.15.78:6789/0 conn(0x7fbd4015e790 legacy=0x7fbd400cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.205+0000 7fbd45aef640 1 -- 172.21.15.46:0/2246595034 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd400d2a40 con 0x7fbd400d1440 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.205+0000 7fbd45aef640 1 --2- 172.21.15.46:0/2246595034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d1440 0x7fbd400b74f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009870 tx=0x7fbd3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8920a24356ab8a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.205+0000 7fbd3ffff640 1 -- 172.21.15.46:0/2246595034 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38034070 con 0x7fbd400d1440 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd3ffff640 1 -- 172.21.15.46:0/2246595034 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd38030840 con 0x7fbd400d1440 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd3ffff640 1 -- 172.21.15.46:0/2246595034 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38030c10 con 0x7fbd400d1440 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 -- 172.21.15.46:0/2246595034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d1440 msgr2=0x7fbd400b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 --2- 172.21.15.46:0/2246595034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d1440 0x7fbd400b74f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009870 tx=0x7fbd3802b980 comp rx=0 tx=0).stop 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 -- 172.21.15.46:0/2246595034 shutdown_connections 2024-04-15T15:17:17.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 --2- 172.21.15.46:0/2246595034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d1440 0x7fbd400b74f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 --2- 172.21.15.46:0/2246595034 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd400d3240 0x7fbd400d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 -- 172.21.15.46:0/2246595034 >> 172.21.15.46:0/2246595034 conn(0x7fbd4008bf60 msgr2=0x7fbd400d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 -- 172.21.15.46:0/2246595034 shutdown_connections 2024-04-15T15:17:17.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.206+0000 7fbd47552640 1 -- 172.21.15.46:0/2246595034 wait complete. 2024-04-15T15:17:17.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.207+0000 7fbd47552640 1 Processor -- start 2024-04-15T15:17:17.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.207+0000 7fbd47552640 1 -- start start 2024-04-15T15:17:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd47552640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd400d1440 0x7fbd400c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd47552640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 0x7fbd4014dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd47552640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd4015e790 0x7fbd401e8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd47552640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400d5180 con 0x7fbd400d3240 2024-04-15T15:17:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd47552640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400d0430 con 0x7fbd400d1440 2024-04-15T15:17:17.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd47552640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbd400d5300 con 0x7fbd4015e790 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd44aed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 0x7fbd4014dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd44aed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 0x7fbd4014dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59870/0 (socket says 172.21.15.46:59870) 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd44aed640 1 -- 172.21.15.46:0/1994705475 learned_addr learned my addr 172.21.15.46:0/1994705475 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd452ee640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd400d1440 0x7fbd400c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.208+0000 7fbd45aef640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd4015e790 0x7fbd401e8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd44aed640 1 -- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd4015e790 msgr2=0x7fbd401e8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd44aed640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd4015e790 0x7fbd401e8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd44aed640 1 -- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd400d1440 msgr2=0x7fbd400c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd44aed640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd400d1440 0x7fbd400c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd44aed640 1 -- 172.21.15.46:0/1994705475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4014e010 con 0x7fbd400d3240 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd44aed640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 0x7fbd4014dad0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400d800 tx=0x7fbd3400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400b6c0 con 0x7fbd400d3240 2024-04-15T15:17:17.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd34019070 con 0x7fbd400d3240 2024-04-15T15:17:17.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd400c9860 con 0x7fbd400d3240 2024-04-15T15:17:17.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400ba80 con 0x7fbd400d3240 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.209+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd400c9e20 con 0x7fbd400d3240 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.210+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd400689e0 con 0x7fbd400d3240 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.211+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fbd3400bc20 con 0x7fbd400d3240 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.211+0000 7fbd3dffb640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbd2c06c980 0x7fbd2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.211+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbd3408c210 con 0x7fbd400d3240 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.211+0000 7fbd452ee640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbd2c06c980 0x7fbd2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.212+0000 7fbd452ee640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbd2c06c980 0x7fbd2c06ee40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbd400a2640 tx=0x7fbd30009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.215+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd4014e010 con 0x7fbd400d3240 2024-04-15T15:17:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.411+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbd400cf660 con 0x7fbd400d3240 2024-04-15T15:17:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.411+0000 7fbd3dffb640 1 -- 172.21.15.46:0/1994705475 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7fbd3405a4c0 con 0x7fbd400d3240 2024-04-15T15:17:17.413 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbd2c06c980 msgr2=0x7fbd2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbd2c06c980 0x7fbd2c06ee40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbd400a2640 tx=0x7fbd30009040 comp rx=0 tx=0).stop 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 msgr2=0x7fbd4014dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 0x7fbd4014dad0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400d800 tx=0x7fbd3400dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 shutdown_connections 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbd2c06c980 0x7fbd2c06ee40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbd4015e790 0x7fbd401e8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd400d3240 0x7fbd4014dad0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 --2- 172.21.15.46:0/1994705475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbd400d1440 0x7fbd400c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 >> 172.21.15.46:0/1994705475 conn(0x7fbd4008bf60 msgr2=0x7fbd4006cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 shutdown_connections 2024-04-15T15:17:17.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.413+0000 7fbd47552640 1 -- 172.21.15.46:0/1994705475 wait complete. 2024-04-15T15:17:17.425 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:17.426 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:17.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.584+0000 7fa5d3888640 1 Processor -- start 2024-04-15T15:17:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.584+0000 7fa5d3888640 1 -- start start 2024-04-15T15:17:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d3888640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5cc09b020 0x7fa5cc0ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d3888640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5cc0d14b0 con 0x7fa5cc09ac40 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d3888640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5cc0cd550 con 0x7fa5cc0b49d0 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d3888640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa5cc0b1b70 con 0x7fa5cc09b020 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d1624640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa5cc09ac40 0x7fa5cc0d0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48892/0 (socket says 172.21.15.46:48892) 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d1e25640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5cc09b020 0x7fa5cc0ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d1624640 1 -- 172.21.15.46:0/596733126 learned_addr learned my addr 172.21.15.46:0/596733126 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d1e25640 1 -- 172.21.15.46:0/596733126 >> v1:172.21.15.78:6789/0 conn(0x7fa5cc0b49d0 legacy=0x7fa5cc160af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d1e25640 1 -- 172.21.15.46:0/596733126 >> v1:172.21.15.46:6789/0 conn(0x7fa5cc09ac40 legacy=0x7fa5cc0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5d1e25640 1 -- 172.21.15.46:0/596733126 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc0cd830 con 0x7fa5cc09b020 2024-04-15T15:17:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.585+0000 7fa5cbfff640 1 -- 172.21.15.46:0/596733126 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1240471681 0 0) 0x7fa5cc0d14b0 con 0x7fa5cc09ac40 2024-04-15T15:17:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.586+0000 7fa5d1e25640 1 --2- 172.21.15.46:0/596733126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0ce030 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009d90 tx=0x7fa5bc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf7930c65d1dfd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.586+0000 7fa5cbfff640 1 -- 172.21.15.46:0/596733126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc039070 con 0x7fa5cc09b020 2024-04-15T15:17:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.586+0000 7fa5cbfff640 1 -- 172.21.15.46:0/596733126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5bc02cd00 con 0x7fa5cc09b020 2024-04-15T15:17:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.586+0000 7fa5cbfff640 1 -- 172.21.15.46:0/596733126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc034660 con 0x7fa5cc09b020 2024-04-15T15:17:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 -- 172.21.15.46:0/596733126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 msgr2=0x7fa5cc0ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 --2- 172.21.15.46:0/596733126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0ce030 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009d90 tx=0x7fa5bc02bf50 comp rx=0 tx=0).stop 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 -- 172.21.15.46:0/596733126 shutdown_connections 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 --2- 172.21.15.46:0/596733126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0ce030 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 -- 172.21.15.46:0/596733126 >> 172.21.15.46:0/596733126 conn(0x7fa5cc0b3eb0 msgr2=0x7fa5cc0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 -- 172.21.15.46:0/596733126 shutdown_connections 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.587+0000 7fa5d3888640 1 -- 172.21.15.46:0/596733126 wait complete. 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.588+0000 7fa5d3888640 1 Processor -- start 2024-04-15T15:17:17.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.588+0000 7fa5d3888640 1 -- start start 2024-04-15T15:17:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d3888640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa5cc09ac40 0x7fa5cc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d3888640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d3888640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5cc0b49d0 0x7fa5cc0b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d3888640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc0ce6f0 con 0x7fa5cc09b020 2024-04-15T15:17:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d3888640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa5cc0ce570 con 0x7fa5cc0b49d0 2024-04-15T15:17:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d3888640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa5cc0c6ba0 con 0x7fa5cc09ac40 2024-04-15T15:17:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59882/0 (socket says 172.21.15.46:59882) 2024-04-15T15:17:17.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 -- 172.21.15.46:0/2330822503 learned_addr learned my addr 172.21.15.46:0/2330822503 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d1624640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa5cc09ac40 0x7fa5cc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d1e25640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5cc0b49d0 0x7fa5cc0b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 -- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa5cc09ac40 msgr2=0x7fa5cc1589c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa5cc09ac40 0x7fa5cc1589c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 -- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5cc0b49d0 msgr2=0x7fa5cc0b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5cc0b49d0 0x7fa5cc0b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 -- 172.21.15.46:0/2330822503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5cc158f00 con 0x7fa5cc09b020 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d1e25640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5cc0b49d0 0x7fa5cc0b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.589+0000 7fa5d0e23640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0b0120 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000b3f0 tx=0x7fa5c000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.590+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0015020 con 0x7fa5cc09b020 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.590+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c0011830 con 0x7fa5cc09b020 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.590+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5cc0cae50 con 0x7fa5cc09b020 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.590+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0011c20 con 0x7fa5cc09b020 2024-04-15T15:17:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.590+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5cc0c6270 con 0x7fa5cc09b020 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.591+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5cc0689e0 con 0x7fa5cc09b020 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.591+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa5c0018020 con 0x7fa5cc09b020 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.592+0000 7fa5c9ffb640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa5b806c980 0x7fa5b806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.592+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa5c008b9b0 con 0x7fa5cc09b020 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.592+0000 7fa5d1624640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa5b806c980 0x7fa5b806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.592+0000 7fa5d1624640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa5b806c980 0x7fa5b806ee40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4000f60 tx=0x7fa5c4007b50 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.596+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5c001f030 con 0x7fa5cc09b020 2024-04-15T15:17:17.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.784+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5cc067bd0 con 0x7fa5cc09b020 2024-04-15T15:17:17.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.785+0000 7fa5c9ffb640 1 -- 172.21.15.46:0/2330822503 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7fa5cc158f00 con 0x7fa5cc09b020 2024-04-15T15:17:17.787 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:17.787 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:02.054907+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:17.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa5b806c980 msgr2=0x7fa5b806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa5b806c980 0x7fa5b806ee40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4000f60 tx=0x7fa5c4007b50 comp rx=0 tx=0).stop 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 msgr2=0x7fa5cc0b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0b0120 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000b3f0 tx=0x7fa5c000b8b0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 shutdown_connections 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa5b806c980 0x7fa5b806ee40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5cc0b49d0 0x7fa5cc0b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5cc09b020 0x7fa5cc0b0120 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 --2- 172.21.15.46:0/2330822503 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa5cc09ac40 0x7fa5cc1589c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.787+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 >> 172.21.15.46:0/2330822503 conn(0x7fa5cc0b3eb0 msgr2=0x7fa5cc0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.788+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 shutdown_connections 2024-04-15T15:17:17.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.788+0000 7fa5d3888640 1 -- 172.21.15.46:0/2330822503 wait complete. 2024-04-15T15:17:17.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:17.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.906+0000 7f014daa7640 1 Processor -- start 2024-04-15T15:17:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.906+0000 7f014daa7640 1 -- start start 2024-04-15T15:17:17.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f014daa7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f014815e790 0x7f01480d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f014daa7640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f01480d1610 0x7f01480d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f014daa7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01480d2480 con 0x7f01480cfd20 2024-04-15T15:17:17.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f014daa7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f01480d2db0 con 0x7f014815e790 2024-04-15T15:17:17.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f014daa7640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f01480d2760 con 0x7f01480d1610 2024-04-15T15:17:17.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f01477fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f014815e790 0x7f01480d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f0146ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f01480cfd20 0x7f01480d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48902/0 (socket says 172.21.15.46:48902) 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f0146ffd640 1 -- 172.21.15.46:0/192384339 learned_addr learned my addr 172.21.15.46:0/192384339 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f0147fff640 1 --2- 172.21.15.46:0/192384339 >> v2:172.21.15.78:3300/0 conn(0x7f01480d1610 0x7f01480d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f01477fe640 1 -- 172.21.15.46:0/192384339 >> v2:172.21.15.78:3300/0 conn(0x7f01480d1610 msgr2=0x7f01480d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f01467fc640 1 -- 172.21.15.46:0/192384339 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221851522 0 0) 0x7f01480d2480 con 0x7f01480cfd20 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f01477fe640 1 --2- 172.21.15.46:0/192384339 >> v2:172.21.15.78:3300/0 conn(0x7f01480d1610 0x7f01480d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f01477fe640 1 -- 172.21.15.46:0/192384339 >> v1:172.21.15.46:6789/0 conn(0x7f01480cfd20 legacy=0x7f01480d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f01477fe640 1 -- 172.21.15.46:0/192384339 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01480d2a40 con 0x7f014815e790 2024-04-15T15:17:17.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.907+0000 7f0147fff640 1 --2- 172.21.15.46:0/192384339 >> v2:172.21.15.78:3300/0 conn(0x7f01480d1610 0x7f01480d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:17.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.908+0000 7f01477fe640 1 --2- 172.21.15.46:0/192384339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01480d3240 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f01380097a0 tx=0x7f013802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d9ecbf5f263003 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.908+0000 7f01467fc640 1 -- 172.21.15.46:0/192384339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138035020 con 0x7f014815e790 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.908+0000 7f01467fc640 1 -- 172.21.15.46:0/192384339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0138031830 con 0x7f014815e790 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.908+0000 7f01467fc640 1 -- 172.21.15.46:0/192384339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138031c00 con 0x7f014815e790 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 -- 172.21.15.46:0/192384339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 msgr2=0x7f01480d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 --2- 172.21.15.46:0/192384339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01480d3240 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f01380097a0 tx=0x7f013802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 -- 172.21.15.46:0/192384339 shutdown_connections 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 --2- 172.21.15.46:0/192384339 >> v2:172.21.15.78:3300/0 conn(0x7f01480d1610 0x7f01480d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 --2- 172.21.15.46:0/192384339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01480d3240 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 -- 172.21.15.46:0/192384339 >> 172.21.15.46:0/192384339 conn(0x7f014808bf60 msgr2=0x7f01480d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 -- 172.21.15.46:0/192384339 shutdown_connections 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.909+0000 7f014daa7640 1 -- 172.21.15.46:0/192384339 wait complete. 2024-04-15T15:17:17.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.910+0000 7f014daa7640 1 Processor -- start 2024-04-15T15:17:17.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.910+0000 7f014daa7640 1 -- start start 2024-04-15T15:17:17.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f014daa7640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01480cfd20 0x7f01481f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f014daa7640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01480d1610 0x7f0148155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f014daa7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01481f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f014daa7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f014815ecf0 con 0x7f014815e790 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f014daa7640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f014815eb70 con 0x7f01480d1610 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f014daa7640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f01480b74f0 con 0x7f01480cfd20 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01481f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01481f3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59900/0 (socket says 172.21.15.46:59900) 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 -- 172.21.15.46:0/1454591363 learned_addr learned my addr 172.21.15.46:0/1454591363 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f01477fe640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01480cfd20 0x7f01481f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0146ffd640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01480d1610 0x7f0148155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 -- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01480cfd20 msgr2=0x7f01481f7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01480cfd20 0x7f01481f7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 -- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01480d1610 msgr2=0x7f0148155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01480d1610 0x7f0148155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0147fff640 1 -- 172.21.15.46:0/1454591363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01481f4830 con 0x7f014815e790 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f01477fe640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01480cfd20 0x7f01481f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.911+0000 7f0146ffd640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01480d1610 0x7f0148155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.912+0000 7f0147fff640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01481f3dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f014000ceb0 tx=0x7f014000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.912+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014001a070 con 0x7f014815e790 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.912+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014000d840 con 0x7f014815e790 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.912+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01481f4b10 con 0x7f014815e790 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.912+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014000dc30 con 0x7f014815e790 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.912+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01480ca860 con 0x7f014815e790 2024-04-15T15:17:17.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.913+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01480689e0 con 0x7f014815e790 2024-04-15T15:17:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.913+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f014001d070 con 0x7f014815e790 2024-04-15T15:17:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.914+0000 7f012ffff640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f013406c980 0x7f013406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.914+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f014008c6b0 con 0x7f014815e790 2024-04-15T15:17:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.914+0000 7f01477fe640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f013406c980 0x7f013406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.914+0000 7f01477fe640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f013406c980 0x7f013406ee40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0138036000 tx=0x7f0138008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:17.918+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f014005a860 con 0x7f014815e790 2024-04-15T15:17:18.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.100+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01480d1ce0 con 0x7f014815e790 2024-04-15T15:17:18.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.101+0000 7f012ffff640 1 -- 172.21.15.46:0/1454591363 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f014005a160 con 0x7f014815e790 2024-04-15T15:17:18.103 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f013406c980 msgr2=0x7f013406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f013406c980 0x7f013406ee40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0138036000 tx=0x7f0138008030 comp rx=0 tx=0).stop 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 msgr2=0x7f01481f3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01481f3dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f014000ceb0 tx=0x7f014000a710 comp rx=0 tx=0).stop 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 shutdown_connections 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f013406c980 0x7f013406ee40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f014815e790 0x7f01481f3dd0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01480d1610 0x7f0148155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 --2- 172.21.15.46:0/1454591363 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01480cfd20 0x7f01481f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 >> 172.21.15.46:0/1454591363 conn(0x7f014808bf60 msgr2=0x7f0148074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 shutdown_connections 2024-04-15T15:17:18.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.103+0000 7f014daa7640 1 -- 172.21.15.46:0/1454591363 wait complete. 2024-04-15T15:17:18.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.222+0000 7f01db167640 1 Processor -- start 2024-04-15T15:17:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.222+0000 7f01db167640 1 -- start start 2024-04-15T15:17:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01db167640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 0x7f01d40d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01db167640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d40d2480 con 0x7f01d40cfd20 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01db167640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d40d2760 con 0x7f01d40d1610 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01db167640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f01d40d2db0 con 0x7f01d415e790 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f01d40cfd20 0x7f01d40d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48906/0 (socket says 172.21.15.46:48906) 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d3fff640 1 -- 172.21.15.46:0/460477936 learned_addr learned my addr 172.21.15.46:0/460477936 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d8f03640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 0x7f01d40d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3844023992 0 0) 0x7f01d40d2480 con 0x7f01d40cfd20 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c4003520 con 0x7f01d40cfd20 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3064099837 0 0) 0x7f01d40d2760 con 0x7f01d40d1610 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d40d2480 con 0x7f01d40d1610 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4244100756 0 0) 0x7f01c4003520 con 0x7f01d40cfd20 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 msgr2=0x7f01d40d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 0x7f01d40d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 >> v1:172.21.15.78:6789/0 conn(0x7f01d40d1610 legacy=0x7f01d40b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d40d2a40 con 0x7f01d40cfd20 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f01c0001670 con 0x7f01d40cfd20 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 >> v1:172.21.15.46:6789/0 conn(0x7f01d40cfd20 legacy=0x7f01d40d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01c4003b70 0x7f01c4003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.223+0000 7f01d37fe640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d37fe640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 0x7f01c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f01d40d2db0 con 0x7f01c4004800 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01c400ba70 con 0x7f01c4003b70 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f01c400bc40 con 0x7f01d415e790 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d3fff640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d8f03640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01c4003b70 0x7f01c4003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d9704640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 0x7f01c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d3fff640 1 -- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 msgr2=0x7f01c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d3fff640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d415e790 0x7f01c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d3fff640 1 -- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01c4003b70 msgr2=0x7f01c4003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d3fff640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01c4003b70 0x7f01c4003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.224+0000 7f01d3fff640 1 -- 172.21.15.46:0/460477936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c4003520 con 0x7f01c4004800 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01d3fff640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01c4004c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f01c000c210 tx=0x7f01c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcf0aa51c4bc00b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c0007ce0 con 0x7f01c4004800 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c0030070 con 0x7f01c4004800 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01d37fe640 1 -- 172.21.15.46:0/460477936 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c0005d50 con 0x7f01c4004800 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01db167640 1 -- 172.21.15.46:0/460477936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 msgr2=0x7f01c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01db167640 1 --2- 172.21.15.46:0/460477936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01c4004c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f01c000c210 tx=0x7f01c4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01d9704640 1 -- 172.21.15.46:0/460477936 reap_dead start 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01db167640 1 -- 172.21.15.46:0/460477936 shutdown_connections 2024-04-15T15:17:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01db167640 1 -- 172.21.15.46:0/460477936 >> 172.21.15.46:0/460477936 conn(0x7f01d408bf60 msgr2=0x7f01d40d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01db167640 1 -- 172.21.15.46:0/460477936 shutdown_connections 2024-04-15T15:17:18.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.225+0000 7f01db167640 1 -- 172.21.15.46:0/460477936 wait complete. 2024-04-15T15:17:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.227+0000 7f01db167640 1 Processor -- start 2024-04-15T15:17:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.227+0000 7f01db167640 1 -- start start 2024-04-15T15:17:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.227+0000 7f01db167640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01d40cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01d40c83e0 0x7f01d4145cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d40c8840 0x7f01d41461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f01d415ecf0 con 0x7f01c4004800 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f01d415eb70 con 0x7f01d40c83e0 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f01d40d0160 con 0x7f01d40c8840 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01d40cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01d40cbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59924/0 (socket says 172.21.15.46:59924) 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 -- 172.21.15.46:0/1372093167 learned_addr learned my addr 172.21.15.46:0/1372093167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d3fff640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01d40c83e0 0x7f01d4145cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d9704640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d40c8840 0x7f01d41461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 -- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d40c8840 msgr2=0x7f01d41461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d40c8840 0x7f01d41461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 -- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01d40c83e0 msgr2=0x7f01d4145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01d40c83e0 0x7f01d4145cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 -- 172.21.15.46:0/1372093167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d40cc4b0 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d8f03640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01d40cbbf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f01c8011910 tx=0x7f01c8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c800faf0 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c800fc90 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d40cc790 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8014440 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.228+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d40ccd50 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.229+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d408c4b0 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.230+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f01c8012040 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.230+0000 7f01d17fa640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f01b806c9b0 0x7f01b806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.230+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f01c808c330 con 0x7f01c4004800 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.230+0000 7f01d3fff640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f01b806c9b0 0x7f01b806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.231+0000 7f01d3fff640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f01b806c9b0 0x7f01b806ee70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f01c0034000 tx=0x7f01c0007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.234+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01c8019030 con 0x7f01c4004800 2024-04-15T15:17:18.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.417+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f01d40689e0 con 0x7f01c4004800 2024-04-15T15:17:18.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.417+0000 7f01d17fa640 1 -- 172.21.15.46:0/1372093167 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f01c805a550 con 0x7f01c4004800 2024-04-15T15:17:18.419 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f01b806c9b0 msgr2=0x7f01b806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f01b806c9b0 0x7f01b806ee70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f01c0034000 tx=0x7f01c0007000 comp rx=0 tx=0).stop 2024-04-15T15:17:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 msgr2=0x7f01d40cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01d40cbbf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f01c8011910 tx=0x7f01c8011dd0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 shutdown_connections 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f01b806c9b0 0x7f01b806ee70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f01d40c8840 0x7f01d41461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f01d40c83e0 0x7f01d4145cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 --2- 172.21.15.46:0/1372093167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01c4004800 0x7f01d40cbbf0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 >> 172.21.15.46:0/1372093167 conn(0x7f01d408bf60 msgr2=0x7f01d40d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 shutdown_connections 2024-04-15T15:17:18.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.419+0000 7f01db167640 1 -- 172.21.15.46:0/1372093167 wait complete. 2024-04-15T15:17:18.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:18.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.538+0000 7fcf1390e640 1 Processor -- start 2024-04-15T15:17:18.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.539+0000 7fcf1390e640 1 -- start start 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.539+0000 7fcf1390e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf0c15e790 0x7fcf0c0d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.539+0000 7fcf1390e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c0d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.539+0000 7fcf1390e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0c0d2480 con 0x7fcf0c0cfd20 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.539+0000 7fcf1390e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fcf0c0d2db0 con 0x7fcf0c15e790 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.539+0000 7fcf1390e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c0d2760 con 0x7fcf0c0d1610 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf0c15e790 0x7fcf0c0d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf0c15e790 0x7fcf0c0d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59932/0 (socket says 172.21.15.46:59932) 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 -- 172.21.15.46:0/1077365655 learned_addr learned my addr 172.21.15.46:0/1077365655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf10ea9640 1 --1- 172.21.15.46:0/1077365655 >> v1:172.21.15.46:6789/0 conn(0x7fcf0c0cfd20 0x7fcf0c0d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48914/0 (socket says 172.21.15.46:48914) 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf11eab640 1 --2- 172.21.15.46:0/1077365655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c0d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 -- 172.21.15.46:0/1077365655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 msgr2=0x7fcf0c0d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 --2- 172.21.15.46:0/1077365655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c0d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 -- 172.21.15.46:0/1077365655 >> v1:172.21.15.46:6789/0 conn(0x7fcf0c0cfd20 legacy=0x7fcf0c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf03fff640 1 -- 172.21.15.46:0/1077365655 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351326365 0 0) 0x7fcf0c0d2480 con 0x7fcf0c0cfd20 2024-04-15T15:17:18.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.540+0000 7fcf116aa640 1 -- 172.21.15.46:0/1077365655 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c0d2a40 con 0x7fcf0c15e790 2024-04-15T15:17:18.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf116aa640 1 --2- 172.21.15.46:0/1077365655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c0d3240 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0d2f30 tx=0x7fcf0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8de409ccb522144 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf03fff640 1 -- 172.21.15.46:0/1077365655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08034070 con 0x7fcf0c15e790 2024-04-15T15:17:18.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf03fff640 1 -- 172.21.15.46:0/1077365655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf08030840 con 0x7fcf0c15e790 2024-04-15T15:17:18.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf03fff640 1 -- 172.21.15.46:0/1077365655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf08030c10 con 0x7fcf0c15e790 2024-04-15T15:17:18.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf1390e640 1 -- 172.21.15.46:0/1077365655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 msgr2=0x7fcf0c0d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1077365655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c0d3240 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0d2f30 tx=0x7fcf0802b980 comp rx=0 tx=0).stop 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf1390e640 1 -- 172.21.15.46:0/1077365655 shutdown_connections 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1077365655 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c0d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1077365655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c0d3240 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.541+0000 7fcf1390e640 1 -- 172.21.15.46:0/1077365655 >> 172.21.15.46:0/1077365655 conn(0x7fcf0c08bf60 msgr2=0x7fcf0c0d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.542+0000 7fcf1390e640 1 -- 172.21.15.46:0/1077365655 shutdown_connections 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.542+0000 7fcf1390e640 1 -- 172.21.15.46:0/1077365655 wait complete. 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 Processor -- start 2024-04-15T15:17:18.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 -- start start 2024-04-15T15:17:18.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c0cfd20 0x7fcf0c159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c15ecf0 con 0x7fcf0c15e790 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c15eb70 con 0x7fcf0c0cfd20 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.543+0000 7fcf1390e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c0b74f0 con 0x7fcf0c0d1610 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c15c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59944/0 (socket says 172.21.15.46:59944) 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 -- 172.21.15.46:0/1195030560 learned_addr learned my addr 172.21.15.46:0/1195030560 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf10ea9640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 -- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 msgr2=0x7fcf0c152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf116aa640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c0cfd20 0x7fcf0c159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 -- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c0cfd20 msgr2=0x7fcf0c159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c0cfd20 0x7fcf0c159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 -- 172.21.15.46:0/1195030560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c159a30 con 0x7fcf0c15e790 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf11eab640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c15c200 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcf04007e40 tx=0x7fcf0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014620 con 0x7fcf0c15e790 2024-04-15T15:17:18.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf04019070 con 0x7fcf0c15e790 2024-04-15T15:17:18.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014a20 con 0x7fcf0c15e790 2024-04-15T15:17:18.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c159d10 con 0x7fcf0c15e790 2024-04-15T15:17:18.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.544+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c15a2d0 con 0x7fcf0c15e790 2024-04-15T15:17:18.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.545+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf0c0689e0 con 0x7fcf0c15e790 2024-04-15T15:17:18.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.546+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcf04014bc0 con 0x7fcf0c15e790 2024-04-15T15:17:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.546+0000 7fcf01ffb640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcef006c9b0 0x7fcef006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.546+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcf0408c610 con 0x7fcf0c15e790 2024-04-15T15:17:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.546+0000 7fcf116aa640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcef006c9b0 0x7fcef006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.547+0000 7fcf116aa640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcef006c9b0 0x7fcef006ee70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcf08036000 tx=0x7fcf08008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.550+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf0405a8c0 con 0x7fcf0c15e790 2024-04-15T15:17:18.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.735+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcf0c0d1ce0 con 0x7fcf0c15e790 2024-04-15T15:17:18.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.735+0000 7fcf01ffb640 1 -- 172.21.15.46:0/1195030560 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fcf0405a1c0 con 0x7fcf0c15e790 2024-04-15T15:17:18.737 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcef006c9b0 msgr2=0x7fcef006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcef006c9b0 0x7fcef006ee70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcf08036000 tx=0x7fcf08008030 comp rx=0 tx=0).stop 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 msgr2=0x7fcf0c15c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c15c200 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcf04007e40 tx=0x7fcf0400e4b0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 shutdown_connections 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcef006c9b0 0x7fcef006ee70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf0c15e790 0x7fcf0c15c200 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcf0c0d1610 0x7fcf0c152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 --2- 172.21.15.46:0/1195030560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf0c0cfd20 0x7fcf0c159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 >> 172.21.15.46:0/1195030560 conn(0x7fcf0c08bf60 msgr2=0x7fcf0c074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 shutdown_connections 2024-04-15T15:17:18.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.737+0000 7fcf1390e640 1 -- 172.21.15.46:0/1195030560 wait complete. 2024-04-15T15:17:18.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.854+0000 7f67f99f4640 1 Processor -- start 2024-04-15T15:17:18.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.854+0000 7f67f99f4640 1 -- start start 2024-04-15T15:17:18.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f99f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f67f40d15e0 0x7f67f40d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f99f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 0x7f67f40d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f99f4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f415eb70 con 0x7f67f415e790 2024-04-15T15:17:18.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f99f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f67f40d3790 con 0x7f67f40d15e0 2024-04-15T15:17:18.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f99f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f67f40d45b0 con 0x7f67f40d4a40 2024-04-15T15:17:18.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f67f415e790 0x7f67f40b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48920/0 (socket says 172.21.15.46:48920) 2024-04-15T15:17:18.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f2ffd640 1 -- 172.21.15.46:0/3700917720 learned_addr learned my addr 172.21.15.46:0/3700917720 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:18.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f27fc640 1 --2- 172.21.15.46:0/3700917720 >> v2:172.21.15.78:3300/0 conn(0x7f67f40d15e0 0x7f67f40d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f37fe640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 0x7f67f40d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1556691274 0 0) 0x7f67f415eb70 con 0x7f67f415e790 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.855+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67e0003520 con 0x7f67f415e790 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 887448488 0 0) 0x7f67e0003520 con 0x7f67f415e790 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 msgr2=0x7f67f40d4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 0x7f67f40d4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 >> v2:172.21.15.78:3300/0 conn(0x7f67f40d15e0 msgr2=0x7f67f40d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 --2- 172.21.15.46:0/3700917720 >> v2:172.21.15.78:3300/0 conn(0x7f67f40d15e0 0x7f67f40d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f40d3c80 con 0x7f67f415e790 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f67e4001a10 con 0x7f67f415e790 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 >> v1:172.21.15.46:6789/0 conn(0x7f67f415e790 legacy=0x7f67f40b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 0x7f67e00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e0003f30 0x7f67e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0004d40 0x7f67e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40d3790 con 0x7f67e0003f30 2024-04-15T15:17:18.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40d45b0 con 0x7f67e0004d40 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f67e000baa0 con 0x7f67f40d4a40 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f27fc640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e0003f30 0x7f67e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f2ffd640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 0x7f67e00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f37fe640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0004d40 0x7f67e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f27fc640 1 -- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 msgr2=0x7f67e00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f27fc640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f40d4a40 0x7f67e00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f27fc640 1 -- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0004d40 msgr2=0x7f67e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f27fc640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0004d40 0x7f67e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.856+0000 7f67f27fc640 1 -- 172.21.15.46:0/3700917720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e0003520 con 0x7f67e0003f30 2024-04-15T15:17:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.857+0000 7f67f27fc640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e0003f30 0x7f67e0004800 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009ac0 tx=0x7f67e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56897f8de005e869 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.857+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc039070 con 0x7f67e0003f30 2024-04-15T15:17:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.857+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67dc02c710 con 0x7f67e0003f30 2024-04-15T15:17:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.857+0000 7f67f1ffb640 1 -- 172.21.15.46:0/3700917720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc009690 con 0x7f67e0003f30 2024-04-15T15:17:18.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.857+0000 7f67f99f4640 1 -- 172.21.15.46:0/3700917720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e0003f30 msgr2=0x7f67e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.857+0000 7f67f99f4640 1 --2- 172.21.15.46:0/3700917720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67e0003f30 0x7f67e0004800 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f67dc009ac0 tx=0x7f67e0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.858+0000 7f67f37fe640 1 -- 172.21.15.46:0/3700917720 reap_dead start 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.858+0000 7f67f99f4640 1 -- 172.21.15.46:0/3700917720 shutdown_connections 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.858+0000 7f67f99f4640 1 -- 172.21.15.46:0/3700917720 >> 172.21.15.46:0/3700917720 conn(0x7f67f4067b40 msgr2=0x7f67f4067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.858+0000 7f67f99f4640 1 -- 172.21.15.46:0/3700917720 shutdown_connections 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.858+0000 7f67f99f4640 1 -- 172.21.15.46:0/3700917720 wait complete. 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.859+0000 7f67f99f4640 1 Processor -- start 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.859+0000 7f67f99f4640 1 -- start start 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f99f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0003f30 0x7f67f41f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f99f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f4158960 0x7f67f41f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f99f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 0x7f67f40cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f99f4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40b7db0 con 0x7f67f40ca770 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f99f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f67f40d4240 con 0x7f67e0003f30 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f99f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f67f40d4eb0 con 0x7f67f4158960 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 0x7f67f40cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 0x7f67f40cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59962/0 (socket says 172.21.15.46:59962) 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 -- 172.21.15.46:0/2513938635 learned_addr learned my addr 172.21.15.46:0/2513938635 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f2ffd640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0003f30 0x7f67f41f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f27fc640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f4158960 0x7f67f41f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 -- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f4158960 msgr2=0x7f67f41f7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f4158960 0x7f67f41f7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 -- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0003f30 msgr2=0x7f67f41f4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0003f30 0x7f67f41f4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 -- 172.21.15.46:0/2513938635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f41f7650 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67f37fe640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 0x7f67f40cec40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f67f41621d0 tx=0x7f67ec011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec004030 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.860+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec01b070 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.861+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f4159610 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.861+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec004430 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.861+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f4159bd0 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.862+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f40689e0 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.862+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f67ec012040 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.862+0000 7f67ebfff640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f67d406c9b0 0x7f67d406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.863+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f67ec08ba70 con 0x7f67f40ca770 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.863+0000 7f67f2ffd640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f67d406c9b0 0x7f67d406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.863+0000 7f67f2ffd640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f67d406c9b0 0x7f67d406ee70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f67e4002230 tx=0x7f67e4006040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:18.867+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67ec059c90 con 0x7f67f40ca770 2024-04-15T15:17:19.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.064+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f67f40c7e80 con 0x7f67f40ca770 2024-04-15T15:17:19.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.065+0000 7f67ebfff640 1 -- 172.21.15.46:0/2513938635 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7f67ec059590 con 0x7f67f40ca770 2024-04-15T15:17:19.067 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:17:19.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f67d406c9b0 msgr2=0x7f67d406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f67d406c9b0 0x7f67d406ee70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f67e4002230 tx=0x7f67e4006040 comp rx=0 tx=0).stop 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 msgr2=0x7f67f40cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 0x7f67f40cec40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f67f41621d0 tx=0x7f67ec011c30 comp rx=0 tx=0).stop 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 shutdown_connections 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f67d406c9b0 0x7f67d406ee70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67f40ca770 0x7f67f40cec40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f67f4158960 0x7f67f41f7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 --2- 172.21.15.46:0/2513938635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67e0003f30 0x7f67f41f4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.066+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 >> 172.21.15.46:0/2513938635 conn(0x7f67f4067b40 msgr2=0x7f67f40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.067+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 shutdown_connections 2024-04-15T15:17:19.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.067+0000 7f67f99f4640 1 -- 172.21.15.46:0/2513938635 wait complete. 2024-04-15T15:17:19.079 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-04-15T15:17:19.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:19.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:19.106 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[54036]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:19.108 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:19.191 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:17:19.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:19.192 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:17:19.207 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:19.207 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:19.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.350+0000 7f85ae159640 1 Processor -- start 2024-04-15T15:17:19.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.350+0000 7f85ae159640 1 -- start start 2024-04-15T15:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85ae159640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85a80d0690 0x7f85a80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85ae159640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f85a80b3160 0x7f85a815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85ae159640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a80d0fb0 con 0x7f85a815ef70 2024-04-15T15:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85ae159640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f85a815cb50 con 0x7f85a80d0690 2024-04-15T15:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85ae159640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f85a815bfd0 con 0x7f85a80b3160 2024-04-15T15:17:19.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85a80d0690 0x7f85a80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f85a80d0690 0x7f85a80d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59966/0 (socket says 172.21.15.46:59966) 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f85a815ef70 0x7f85a815f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:48922/0 (socket says 172.21.15.46:48922) 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 -- 172.21.15.46:0/2597188306 learned_addr learned my addr 172.21.15.46:0/2597188306 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a6ffd640 1 --2- 172.21.15.46:0/2597188306 >> v2:172.21.15.78:3300/0 conn(0x7f85a80b3160 0x7f85a815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 -- 172.21.15.46:0/2597188306 >> v2:172.21.15.78:3300/0 conn(0x7f85a80b3160 msgr2=0x7f85a815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 --2- 172.21.15.46:0/2597188306 >> v2:172.21.15.78:3300/0 conn(0x7f85a80b3160 0x7f85a815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 -- 172.21.15.46:0/2597188306 >> v1:172.21.15.46:6789/0 conn(0x7f85a815ef70 legacy=0x7f85a815f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.351+0000 7f85a77fe640 1 -- 172.21.15.46:0/2597188306 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a815c2b0 con 0x7f85a80d0690 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.352+0000 7f85a77fe640 1 --2- 172.21.15.46:0/2597188306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80d0690 0x7f85a80d0a70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f85a806fec0 tx=0x7f859802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e44d9151afe7a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.352+0000 7f85a67fc640 1 -- 172.21.15.46:0/2597188306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598034070 con 0x7f85a80d0690 2024-04-15T15:17:19.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.352+0000 7f85a67fc640 1 -- 172.21.15.46:0/2597188306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8598030860 con 0x7f85a80d0690 2024-04-15T15:17:19.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.352+0000 7f85a67fc640 1 -- 172.21.15.46:0/2597188306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598030c30 con 0x7f85a80d0690 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 -- 172.21.15.46:0/2597188306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80d0690 msgr2=0x7f85a80d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 --2- 172.21.15.46:0/2597188306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80d0690 0x7f85a80d0a70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f85a806fec0 tx=0x7f859802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 -- 172.21.15.46:0/2597188306 shutdown_connections 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 --2- 172.21.15.46:0/2597188306 >> v2:172.21.15.78:3300/0 conn(0x7f85a80b3160 0x7f85a815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 --2- 172.21.15.46:0/2597188306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80d0690 0x7f85a80d0a70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 -- 172.21.15.46:0/2597188306 >> 172.21.15.46:0/2597188306 conn(0x7f85a806be00 msgr2=0x7f85a806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 -- 172.21.15.46:0/2597188306 shutdown_connections 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.353+0000 7f85ae159640 1 -- 172.21.15.46:0/2597188306 wait complete. 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.354+0000 7f85ae159640 1 Processor -- start 2024-04-15T15:17:19.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.354+0000 7f85ae159640 1 -- start start 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85ae159640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 0x7f85a80cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85ae159640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85a80d0690 0x7f85a80aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85ae159640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a815ef70 0x7f85a809a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85ae159640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85a80b4fb0 con 0x7f85a80b3160 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85ae159640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f85a815c870 con 0x7f85a815ef70 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85ae159640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f85a815bbd0 con 0x7f85a80d0690 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 0x7f85a80cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 0x7f85a80cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59970/0 (socket says 172.21.15.46:59970) 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85a77fe640 1 -- 172.21.15.46:0/3150498737 learned_addr learned my addr 172.21.15.46:0/3150498737 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.355+0000 7f85a6ffd640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85a80d0690 0x7f85a80aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a7fff640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a815ef70 0x7f85a809a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a77fe640 1 -- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85a80d0690 msgr2=0x7f85a80aee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a77fe640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85a80d0690 0x7f85a80aee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a77fe640 1 -- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a815ef70 msgr2=0x7f85a809a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a77fe640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a815ef70 0x7f85a809a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a77fe640 1 -- 172.21.15.46:0/3150498737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a814f2e0 con 0x7f85a80b3160 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a7fff640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a815ef70 0x7f85a809a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85a77fe640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 0x7f85a80cf300 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f859802b9e0 tx=0x7f85980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598034040 con 0x7f85a80b3160 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8598039d90 con 0x7f85a80b3160 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a814f5c0 con 0x7f85a80b3160 2024-04-15T15:17:19.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598043610 con 0x7f85a80b3160 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.356+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a814fb80 con 0x7f85a80b3160 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.357+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a80689e0 con 0x7f85a80b3160 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.357+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f859804a060 con 0x7f85a80b3160 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.358+0000 7f858ffff640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f859406c9b0 0x7f859406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.358+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f85980adeb0 con 0x7f85a80b3160 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.358+0000 7f85a6ffd640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f859406c9b0 0x7f859406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.359+0000 7f85a6ffd640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f859406c9b0 0x7f859406ee70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f859c006fd0 tx=0x7f859c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.362+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f859803e070 con 0x7f85a80b3160 2024-04-15T15:17:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.549+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85a80b18f0 con 0x7f85a80b3160 2024-04-15T15:17:19.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.550+0000 7f858ffff640 1 -- 172.21.15.46:0/3150498737 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16894 (secure 0 0 0) 0x7f8598037360 con 0x7f85a80b3160 2024-04-15T15:17:19.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:19.553 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":39,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:02.054907+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f859406c9b0 msgr2=0x7f859406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f859406c9b0 0x7f859406ee70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f859c006fd0 tx=0x7f859c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 msgr2=0x7f85a80cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 0x7f85a80cf300 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f859802b9e0 tx=0x7f85980046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 shutdown_connections 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f859406c9b0 0x7f859406ee70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85a815ef70 0x7f85a809a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85a80d0690 0x7f85a80aee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 --2- 172.21.15.46:0/3150498737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85a80b3160 0x7f85a80cf300 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 >> 172.21.15.46:0/3150498737 conn(0x7f85a806be00 msgr2=0x7f85a80ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 shutdown_connections 2024-04-15T15:17:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.553+0000 7f85ae159640 1 -- 172.21.15.46:0/3150498737 wait complete. 2024-04-15T15:17:19.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:19.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.672+0000 7fbcfe0f2640 1 Processor -- start 2024-04-15T15:17:19.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.672+0000 7fbcfe0f2640 1 -- start start 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcfe0f2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbcf809a9d0 0x7fbcf8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcfe0f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf80701f0 0x7fbcf815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcfe0f2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf8070720 con 0x7fbcf809adb0 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcfe0f2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbcf809ba90 con 0x7fbcf809a9d0 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcfe0f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf8160070 con 0x7fbcf80701f0 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbcf809a9d0 0x7fbcf8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbcf809adb0 0x7fbcf80d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55856/0 (socket says 172.21.15.46:55856) 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf7fff640 1 -- 172.21.15.46:0/3172174937 learned_addr learned my addr 172.21.15.46:0/3172174937 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf6ffd640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf80701f0 0x7fbcf815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf77fe640 1 -- 172.21.15.46:0/3172174937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf80701f0 msgr2=0x7fbcf815fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf77fe640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf80701f0 0x7fbcf815fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf77fe640 1 -- 172.21.15.46:0/3172174937 >> v1:172.21.15.46:6789/0 conn(0x7fbcf809adb0 legacy=0x7fbcf80d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf77fe640 1 -- 172.21.15.46:0/3172174937 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf8070a90 con 0x7fbcf809a9d0 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf67fc640 1 -- 172.21.15.46:0/3172174937 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2394601048 0 0) 0x7fbcf8070720 con 0x7fbcf809adb0 2024-04-15T15:17:19.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.673+0000 7fbcf6ffd640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf80701f0 0x7fbcf815fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.674+0000 7fbcf77fe640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf809a9d0 0x7fbcf8160420 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbce80097d0 tx=0x7fbce802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2688c7032194047 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.674+0000 7fbcf67fc640 1 -- 172.21.15.46:0/3172174937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8035020 con 0x7fbcf809a9d0 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.674+0000 7fbcf67fc640 1 -- 172.21.15.46:0/3172174937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce8031800 con 0x7fbcf809a9d0 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.674+0000 7fbcf67fc640 1 -- 172.21.15.46:0/3172174937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8031bd0 con 0x7fbcf809a9d0 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/3172174937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf809a9d0 msgr2=0x7fbcf8160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf809a9d0 0x7fbcf8160420 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbce80097d0 tx=0x7fbce802b8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/3172174937 shutdown_connections 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf80701f0 0x7fbcf815fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/3172174937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf809a9d0 0x7fbcf8160420 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/3172174937 >> 172.21.15.46:0/3172174937 conn(0x7fbcf80d2460 msgr2=0x7fbcf80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/3172174937 shutdown_connections 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.675+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/3172174937 wait complete. 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.676+0000 7fbcfe0f2640 1 Processor -- start 2024-04-15T15:17:19.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.676+0000 7fbcfe0f2640 1 -- start start 2024-04-15T15:17:19.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcfe0f2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 0x7fbcf806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcfe0f2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf809a9d0 0x7fbcf815a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcfe0f2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcf809adb0 0x7fbcf80b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcfe0f2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf8071050 con 0x7fbcf80701f0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcfe0f2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf80d08c0 con 0x7fbcf809adb0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcfe0f2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf80b1c60 con 0x7fbcf809a9d0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 0x7fbcf806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 0x7fbcf806d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60542/0 (socket says 172.21.15.46:60542) 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 -- 172.21.15.46:0/2236941319 learned_addr learned my addr 172.21.15.46:0/2236941319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf6ffd640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf809a9d0 0x7fbcf815a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf7fff640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcf809adb0 0x7fbcf80b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 -- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf809a9d0 msgr2=0x7fbcf815a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf809a9d0 0x7fbcf815a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 -- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcf809adb0 msgr2=0x7fbcf80b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcf809adb0 0x7fbcf80b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf77fe640 1 -- 172.21.15.46:0/2236941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf806daf0 con 0x7fbcf80701f0 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.677+0000 7fbcf7fff640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcf809adb0 0x7fbcf80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:19.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.678+0000 7fbcf77fe640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 0x7fbcf806d410 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbce8000c00 tx=0x7fbce80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.678+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8041070 con 0x7fbcf80701f0 2024-04-15T15:17:19.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.678+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce803ad00 con 0x7fbcf80701f0 2024-04-15T15:17:19.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.678+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf81f7a90 con 0x7fbcf80701f0 2024-04-15T15:17:19.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.678+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce8035040 con 0x7fbcf80701f0 2024-04-15T15:17:19.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.678+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf81f8330 con 0x7fbcf80701f0 2024-04-15T15:17:19.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.679+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf81f7c60 con 0x7fbcf80701f0 2024-04-15T15:17:19.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.679+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fbce8038020 con 0x7fbcf80701f0 2024-04-15T15:17:19.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.680+0000 7fbcdffff640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbce406c980 0x7fbce406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.680+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbce80ada60 con 0x7fbcf80701f0 2024-04-15T15:17:19.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.680+0000 7fbcf6ffd640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbce406c980 0x7fbce406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.681+0000 7fbcf6ffd640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbce406c980 0x7fbce406ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80cf4f0 tx=0x7fbcec008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.684+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbce8043350 con 0x7fbcf80701f0 2024-04-15T15:17:19.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.872+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbcf80689e0 con 0x7fbcf80701f0 2024-04-15T15:17:19.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.872+0000 7fbcdffff640 1 -- 172.21.15.46:0/2236941319 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7fbce807bd10 con 0x7fbcf80701f0 2024-04-15T15:17:19.874 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:19.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbce406c980 msgr2=0x7fbce406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbce406c980 0x7fbce406ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80cf4f0 tx=0x7fbcec008040 comp rx=0 tx=0).stop 2024-04-15T15:17:19.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 msgr2=0x7fbcf806d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 0x7fbcf806d410 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fbce8000c00 tx=0x7fbce80046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 shutdown_connections 2024-04-15T15:17:19.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbce406c980 0x7fbce406ee40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbcf809adb0 0x7fbcf80b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbcf809a9d0 0x7fbcf815a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 --2- 172.21.15.46:0/2236941319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcf80701f0 0x7fbcf806d410 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 >> 172.21.15.46:0/2236941319 conn(0x7fbcf80d2460 msgr2=0x7fbcf815df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 shutdown_connections 2024-04-15T15:17:19.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.874+0000 7fbcfe0f2640 1 -- 172.21.15.46:0/2236941319 wait complete. 2024-04-15T15:17:19.887 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:19.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.995+0000 7fb78862a640 1 Processor -- start 2024-04-15T15:17:19.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.995+0000 7fb78862a640 1 -- start start 2024-04-15T15:17:19.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb78862a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb78015e790 0x7fb7800d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb78862a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb78862a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7800d2480 con 0x7fb7800cfd20 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb78862a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb7800d2db0 con 0x7fb78015e790 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb78862a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb7800d2760 con 0x7fb7800d1610 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb78015e790 0x7fb7800d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb78015e790 0x7fb7800d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60544/0 (socket says 172.21.15.46:60544) 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 -- 172.21.15.46:0/4093932937 learned_addr learned my addr 172.21.15.46:0/4093932937 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb786bc7640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 -- 172.21.15.46:0/4093932937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 msgr2=0x7fb7800d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 -- 172.21.15.46:0/4093932937 >> v1:172.21.15.78:6789/0 conn(0x7fb7800cfd20 legacy=0x7fb7800d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb7863c6640 1 -- 172.21.15.46:0/4093932937 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7800d2a40 con 0x7fb78015e790 2024-04-15T15:17:19.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.996+0000 7fb786bc7640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.997+0000 7fb7863c6640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb78015e790 0x7fb7800d3240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb7800d2f30 tx=0x7fb77c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9682188c1464c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.997+0000 7fb7853c4640 1 -- 172.21.15.46:0/4093932937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c02ca50 con 0x7fb78015e790 2024-04-15T15:17:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.997+0000 7fb7853c4640 1 -- 172.21.15.46:0/4093932937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77c02cbf0 con 0x7fb78015e790 2024-04-15T15:17:20.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.997+0000 7fb7853c4640 1 -- 172.21.15.46:0/4093932937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0343f0 con 0x7fb78015e790 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 -- 172.21.15.46:0/4093932937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb78015e790 msgr2=0x7fb7800d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb78015e790 0x7fb7800d3240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fb7800d2f30 tx=0x7fb77c02beb0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 -- 172.21.15.46:0/4093932937 shutdown_connections 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 --2- 172.21.15.46:0/4093932937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb78015e790 0x7fb7800d3240 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 -- 172.21.15.46:0/4093932937 >> 172.21.15.46:0/4093932937 conn(0x7fb78008bf60 msgr2=0x7fb7800d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 -- 172.21.15.46:0/4093932937 shutdown_connections 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.998+0000 7fb78862a640 1 -- 172.21.15.46:0/4093932937 wait complete. 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.999+0000 7fb78862a640 1 Processor -- start 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:19.999+0000 7fb78862a640 1 -- start start 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb78862a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 0x7fb780158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb78862a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800cad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb78862a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb78015e790 0x7fb78015cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb78862a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb78015ecf0 con 0x7fb7800cfd20 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb78862a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb78015eb70 con 0x7fb78015e790 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb78862a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb7800b74f0 con 0x7fb7800d1610 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 0x7fb780158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 0x7fb780158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60558/0 (socket says 172.21.15.46:60558) 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 -- 172.21.15.46:0/1374234282 learned_addr learned my addr 172.21.15.46:0/1374234282 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb786bc7640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb78015e790 0x7fb78015cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb785bc5640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800cad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 -- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 msgr2=0x7fb7800cad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800cad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 -- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb78015e790 msgr2=0x7fb78015cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb78015e790 0x7fb78015cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 -- 172.21.15.46:0/1374234282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7801598a0 con 0x7fb7800cfd20 2024-04-15T15:17:20.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb785bc5640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb7863c6640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 0x7fb780158da0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb77c036040 tx=0x7fb77c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.000+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c043020 con 0x7fb7800cfd20 2024-04-15T15:17:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.001+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77c03e450 con 0x7fb7800cfd20 2024-04-15T15:17:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.001+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb780159b80 con 0x7fb7800cfd20 2024-04-15T15:17:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.001+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c009510 con 0x7fb7800cfd20 2024-04-15T15:17:20.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.001+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78015a140 con 0x7fb7800cfd20 2024-04-15T15:17:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.002+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7800689e0 con 0x7fb7800cfd20 2024-04-15T15:17:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.002+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb77c046020 con 0x7fb7800cfd20 2024-04-15T15:17:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.002+0000 7fb776ffd640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb76406c980 0x7fb76406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.002+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7fb77c0ae5e0 con 0x7fb7800cfd20 2024-04-15T15:17:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.003+0000 7fb785bc5640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb76406c980 0x7fb76406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.003+0000 7fb785bc5640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb76406c980 0x7fb76406ee40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb770003eb0 tx=0x7fb770009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.007+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb77c04d030 con 0x7fb7800cfd20 2024-04-15T15:17:20.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.189+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb7800ce450 con 0x7fb7800cfd20 2024-04-15T15:17:20.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.189+0000 7fb776ffd640 1 -- 172.21.15.46:0/1374234282 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) ==== 104+0+11 (secure 0 0 0) 0x7fb77c07c900 con 0x7fb7800cfd20 2024-04-15T15:17:20.191 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb76406c980 msgr2=0x7fb76406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb76406c980 0x7fb76406ee40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb770003eb0 tx=0x7fb770009040 comp rx=0 tx=0).stop 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 msgr2=0x7fb780158da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 0x7fb780158da0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb77c036040 tx=0x7fb77c004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 shutdown_connections 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb76406c980 0x7fb76406ee40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb78015e790 0x7fb78015cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb7800d1610 0x7fb7800cad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 --2- 172.21.15.46:0/1374234282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7800cfd20 0x7fb780158da0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 >> 172.21.15.46:0/1374234282 conn(0x7fb78008bf60 msgr2=0x7fb780074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 shutdown_connections 2024-04-15T15:17:20.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.191+0000 7fb78862a640 1 -- 172.21.15.46:0/1374234282 wait complete. 2024-04-15T15:17:20.204 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.310+0000 7fb91eb32640 1 Processor -- start 2024-04-15T15:17:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.311+0000 7fb91eb32640 1 -- start start 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.311+0000 7fb91eb32640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb91809b0e0 0x7fb9180cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.311+0000 7fb91eb32640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9180d0350 con 0x7fb91809b650 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.311+0000 7fb91eb32640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9180ce100 con 0x7fb91815c8d0 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.311+0000 7fb91eb32640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb9180b5b00 con 0x7fb91809b0e0 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.311+0000 7fb91d0cf640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb91809b650 0x7fb9180b5f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55862/0 (socket says 172.21.15.46:55862) 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb91d0cf640 1 -- 172.21.15.46:0/764630002 learned_addr learned my addr 172.21.15.46:0/764630002 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb917fff640 1 --2- 172.21.15.46:0/764630002 >> v2:172.21.15.78:3300/0 conn(0x7fb91809b0e0 0x7fb9180cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb9177fe640 1 -- 172.21.15.46:0/764630002 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3154950190 0 0) 0x7fb9180d0350 con 0x7fb91809b650 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb9177fe640 1 -- 172.21.15.46:0/764630002 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb904003520 con 0x7fb91809b650 2024-04-15T15:17:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb917fff640 1 -- 172.21.15.46:0/764630002 >> v1:172.21.15.78:6789/0 conn(0x7fb91815c8d0 legacy=0x7fb91809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb917fff640 1 -- 172.21.15.46:0/764630002 >> v1:172.21.15.46:6789/0 conn(0x7fb91809b650 legacy=0x7fb9180b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.312+0000 7fb917fff640 1 -- 172.21.15.46:0/764630002 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9180b54b0 con 0x7fb91809b0e0 2024-04-15T15:17:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.313+0000 7fb917fff640 1 --2- 172.21.15.46:0/764630002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb91809b0e0 0x7fb9180cda70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb9000097a0 tx=0x7fb90002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed1ef59ce2a61bbf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.313+0000 7fb9177fe640 1 -- 172.21.15.46:0/764630002 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900034070 con 0x7fb91809b0e0 2024-04-15T15:17:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.313+0000 7fb9177fe640 1 -- 172.21.15.46:0/764630002 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb900030610 con 0x7fb91809b0e0 2024-04-15T15:17:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.313+0000 7fb9177fe640 1 -- 172.21.15.46:0/764630002 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9000309c0 con 0x7fb91809b0e0 2024-04-15T15:17:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.313+0000 7fb91eb32640 1 -- 172.21.15.46:0/764630002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb91809b0e0 msgr2=0x7fb9180cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.313+0000 7fb91eb32640 1 --2- 172.21.15.46:0/764630002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb91809b0e0 0x7fb9180cda70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb9000097a0 tx=0x7fb90002b960 comp rx=0 tx=0).stop 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.314+0000 7fb91eb32640 1 -- 172.21.15.46:0/764630002 shutdown_connections 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.314+0000 7fb91eb32640 1 --2- 172.21.15.46:0/764630002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb91809b0e0 0x7fb9180cda70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.314+0000 7fb91eb32640 1 -- 172.21.15.46:0/764630002 >> 172.21.15.46:0/764630002 conn(0x7fb9180b3fb0 msgr2=0x7fb9180b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.314+0000 7fb91eb32640 1 -- 172.21.15.46:0/764630002 shutdown_connections 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.314+0000 7fb91eb32640 1 -- 172.21.15.46:0/764630002 wait complete. 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.315+0000 7fb91eb32640 1 Processor -- start 2024-04-15T15:17:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.315+0000 7fb91eb32640 1 -- start start 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 0x7fb9181e6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb91809b650 0x7fb91815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9181e67f0 0x7fb9181e6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb9180b6820 con 0x7fb91809b0e0 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb9180b66a0 con 0x7fb9181e67f0 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb9180b69a0 con 0x7fb91809b650 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 0x7fb9181e6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 0x7fb9181e6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60566/0 (socket says 172.21.15.46:60566) 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 -- 172.21.15.46:0/1620025021 learned_addr learned my addr 172.21.15.46:0/1620025021 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb917fff640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb91809b650 0x7fb91815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91d0cf640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9181e67f0 0x7fb9181e6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 -- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb91809b650 msgr2=0x7fb91815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb91809b650 0x7fb91815a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 -- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9181e67f0 msgr2=0x7fb9181e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9181e67f0 0x7fb9181e6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 -- 172.21.15.46:0/1620025021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91815a8a0 con 0x7fb91809b0e0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91d0cf640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9181e67f0 0x7fb9181e6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91c8ce640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 0x7fb9181e6010 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c3f0 tx=0x7fb91000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9100052d0 con 0x7fb91809b0e0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb910005490 con 0x7fb91809b0e0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9180cbcd0 con 0x7fb91809b0e0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910007260 con 0x7fb91809b0e0 2024-04-15T15:17:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.316+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91806d320 con 0x7fb91809b0e0 2024-04-15T15:17:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.318+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9180690f0 con 0x7fb91809b0e0 2024-04-15T15:17:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.318+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb910007400 con 0x7fb91809b0e0 2024-04-15T15:17:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.318+0000 7fb9157fa640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb8f806c980 0x7fb8f806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.318+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7fb91008b2b0 con 0x7fb91809b0e0 2024-04-15T15:17:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.319+0000 7fb917fff640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb8f806c980 0x7fb8f806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.319+0000 7fb917fff640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb8f806c980 0x7fb8f806ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb900036000 tx=0x7fb900008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.323+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb910058de0 con 0x7fb91809b0e0 2024-04-15T15:17:20.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.508+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb9180cecb0 con 0x7fb91809b0e0 2024-04-15T15:17:20.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.508+0000 7fb9157fa640 1 -- 172.21.15.46:0/1620025021 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) ==== 100+0+11 (secure 0 0 0) 0x7fb91005d0e0 con 0x7fb91809b0e0 2024-04-15T15:17:20.510 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:20.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb8f806c980 msgr2=0x7fb8f806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb8f806c980 0x7fb8f806ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb900036000 tx=0x7fb900008030 comp rx=0 tx=0).stop 2024-04-15T15:17:20.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 msgr2=0x7fb9181e6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 0x7fb9181e6010 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb91000c3f0 tx=0x7fb91000c8b0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 shutdown_connections 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb8f806c980 0x7fb8f806ee40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9181e67f0 0x7fb9181e6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb91809b650 0x7fb91815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 --2- 172.21.15.46:0/1620025021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb91809b0e0 0x7fb9181e6010 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 >> 172.21.15.46:0/1620025021 conn(0x7fb9180b3fb0 msgr2=0x7fb9180710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 shutdown_connections 2024-04-15T15:17:20.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.510+0000 7fb91eb32640 1 -- 172.21.15.46:0/1620025021 wait complete. 2024-04-15T15:17:20.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.627+0000 7f109c6d9640 1 Processor -- start 2024-04-15T15:17:20.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.627+0000 7f109c6d9640 1 -- start start 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109c6d9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 0x7f10940d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109c6d9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f10940b3160 0x7f109415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109c6d9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10940d0fb0 con 0x7f109415ef70 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109c6d9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f109415cb50 con 0x7f10940b3160 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109c6d9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f109415bfd0 con 0x7f10940d0690 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109ac76640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f109415ef70 0x7f109415f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55870/0 (socket says 172.21.15.46:55870) 2024-04-15T15:17:20.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109ac76640 1 -- 172.21.15.46:0/518152417 learned_addr learned my addr 172.21.15.46:0/518152417 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f109a475640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 0x7f10940d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f1099c74640 1 --2- 172.21.15.46:0/518152417 >> v2:172.21.15.78:3300/0 conn(0x7f10940b3160 0x7f109415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2791334968 0 0) 0x7f10940d0fb0 con 0x7f109415ef70 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.628+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1080003520 con 0x7f109415ef70 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3340068503 0 0) 0x7f1080003520 con 0x7f109415ef70 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 msgr2=0x7f10940d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 0x7f10940d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 >> v2:172.21.15.78:3300/0 conn(0x7f10940b3160 msgr2=0x7f109415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 --2- 172.21.15.46:0/518152417 >> v2:172.21.15.78:3300/0 conn(0x7f10940b3160 0x7f109415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109415c2b0 con 0x7f109415ef70 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f108c0019f0 con 0x7f109415ef70 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 >> v1:172.21.15.46:6789/0 conn(0x7f109415ef70 legacy=0x7f109415f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 0x7f10800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080003f30 0x7f1080004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1080004d40 0x7f10800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f109415cb50 con 0x7f1080004d40 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f109415bfd0 con 0x7f1080003f30 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f108000dbd0 con 0x7f10940d0690 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109ac76640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1080004d40 0x7f10800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109a475640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 0x7f10800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f1099c74640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080003f30 0x7f1080004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109ac76640 1 -- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 msgr2=0x7f10800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109ac76640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10940d0690 0x7f10800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109ac76640 1 -- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080003f30 msgr2=0x7f1080004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109ac76640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080003f30 0x7f1080004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.629+0000 7f109ac76640 1 -- 172.21.15.46:0/518152417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1080003520 con 0x7f1080004d40 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.630+0000 7f109ac76640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1080004d40 0x7f10800051c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f108c00a920 tx=0x7f1080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0fdcf8d6a96dead server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.630+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c00a4c0 con 0x7f1080004d40 2024-04-15T15:17:20.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.630+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108c034650 con 0x7f1080004d40 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.630+0000 7f1099473640 1 -- 172.21.15.46:0/518152417 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c034a80 con 0x7f1080004d40 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109c6d9640 1 -- 172.21.15.46:0/518152417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1080004d40 msgr2=0x7f10800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109c6d9640 1 --2- 172.21.15.46:0/518152417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1080004d40 0x7f10800051c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f108c00a920 tx=0x7f1080005830 comp rx=0 tx=0).stop 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109ac76640 1 -- 172.21.15.46:0/518152417 reap_dead start 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109c6d9640 1 -- 172.21.15.46:0/518152417 shutdown_connections 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109c6d9640 1 -- 172.21.15.46:0/518152417 >> 172.21.15.46:0/518152417 conn(0x7f109406be00 msgr2=0x7f109406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109c6d9640 1 -- 172.21.15.46:0/518152417 shutdown_connections 2024-04-15T15:17:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.631+0000 7f109c6d9640 1 -- 172.21.15.46:0/518152417 wait complete. 2024-04-15T15:17:20.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.632+0000 7f109c6d9640 1 Processor -- start 2024-04-15T15:17:20.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.632+0000 7f109c6d9640 1 -- start start 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109c6d9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080004d40 0x7f109415a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109c6d9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1094147980 0x7f1094159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109c6d9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 0x7f10941e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109c6d9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10940b4fb0 con 0x7f10940cddc0 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109c6d9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f109415c870 con 0x7f1080004d40 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109c6d9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f109415bbd0 con 0x7f1094147980 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 0x7f10941e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 0x7f10941e5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60578/0 (socket says 172.21.15.46:60578) 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 -- 172.21.15.46:0/2361570846 learned_addr learned my addr 172.21.15.46:0/2361570846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109a475640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080004d40 0x7f109415a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f1099c74640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1094147980 0x7f1094159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 -- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1094147980 msgr2=0x7f1094159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1094147980 0x7f1094159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 -- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080004d40 msgr2=0x7f109415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080004d40 0x7f109415a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 -- 172.21.15.46:0/2361570846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1094159cd0 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f109ac76640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 0x7f10941e5de0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f108c00a920 tx=0x7f109415c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.633+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c002000 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.634+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f108c044070 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.634+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10940ccd20 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.634+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108c034a00 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.634+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10940cd2e0 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.635+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10940689e0 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.635+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f108c034ba0 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.635+0000 7f108affd640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f107806c980 0x7f107806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.635+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f108c0ae190 con 0x7f10940cddc0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.636+0000 7f109a475640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f107806c980 0x7f107806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.636+0000 7f109a475640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f107806c980 0x7f107806ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1090009bc0 tx=0x7f1090009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.640+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1094159cd0 con 0x7f10940cddc0 2024-04-15T15:17:20.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.840+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1094159ee0 con 0x7f10940cddc0 2024-04-15T15:17:20.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.840+0000 7f108affd640 1 -- 172.21.15.46:0/2361570846 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7f108c07c240 con 0x7f10940cddc0 2024-04-15T15:17:20.842 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:17:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f107806c980 msgr2=0x7f107806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f107806c980 0x7f107806ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1090009bc0 tx=0x7f1090009190 comp rx=0 tx=0).stop 2024-04-15T15:17:20.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 msgr2=0x7f10941e5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 0x7f10941e5de0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f108c00a920 tx=0x7f109415c480 comp rx=0 tx=0).stop 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 shutdown_connections 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f107806c980 0x7f107806ee40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10940cddc0 0x7f10941e5de0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f1094147980 0x7f1094159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 --2- 172.21.15.46:0/2361570846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1080004d40 0x7f109415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 >> 172.21.15.46:0/2361570846 conn(0x7f109406be00 msgr2=0x7f10940b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 shutdown_connections 2024-04-15T15:17:20.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.842+0000 7f109c6d9640 1 -- 172.21.15.46:0/2361570846 wait complete. 2024-04-15T15:17:20.855 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:20.855 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:20.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.980+0000 7f540fe63640 1 Processor -- start 2024-04-15T15:17:20.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.980+0000 7f540fe63640 1 -- start start 2024-04-15T15:17:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.981+0000 7f540fe63640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54080d15e0 0x7f54080d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.981+0000 7f540fe63640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540815eb70 con 0x7f540815e790 2024-04-15T15:17:20.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.981+0000 7f540fe63640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54080d3c80 con 0x7f54080d4a40 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.981+0000 7f540fe63640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f54080d3790 con 0x7f54080d15e0 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.981+0000 7f540dbff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f540815e790 0x7f54080b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55876/0 (socket says 172.21.15.46:55876) 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.981+0000 7f540dbff640 1 -- 172.21.15.46:0/3476027877 learned_addr learned my addr 172.21.15.46:0/3476027877 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429561466 0 0) 0x7f540815eb70 con 0x7f540815e790 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 --2- 172.21.15.46:0/3476027877 >> v2:172.21.15.78:3300/0 conn(0x7f54080d15e0 0x7f54080d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f4003520 con 0x7f540815e790 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1556852882 0 0) 0x7f53f4003520 con 0x7f540815e790 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 >> v2:172.21.15.78:3300/0 conn(0x7f54080d15e0 msgr2=0x7f54080d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 --2- 172.21.15.46:0/3476027877 >> v2:172.21.15.78:3300/0 conn(0x7f54080d15e0 0x7f54080d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 >> v1:172.21.15.78:6789/0 conn(0x7f54080d4a40 legacy=0x7f54080d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54080d3f60 con 0x7f540815e790 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f5404001690 con 0x7f540815e790 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 >> v1:172.21.15.46:6789/0 conn(0x7f540815e790 legacy=0x7f54080b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f40039b0 0x7f53f4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f4004800 0x7f53f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53f40052e0 0x7f53f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54080d3790 con 0x7f53f4004800 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f53f400c1b0 con 0x7f53f40052e0 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f53f400c3b0 con 0x7f53f40039b0 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f4004800 0x7f53f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540e400640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53f40052e0 0x7f53f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540dbff640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f40039b0 0x7f53f4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 -- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f40039b0 msgr2=0x7f53f4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f40039b0 0x7f53f4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 -- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53f40052e0 msgr2=0x7f53f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53f40052e0 0x7f53f400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540d3fe640 1 -- 172.21.15.46:0/3476027877 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f4003520 con 0x7f53f4004800 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540e400640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53f40052e0 0x7f53f400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:20.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.982+0000 7f540e400640 1 -- 172.21.15.46:0/3476027877 reap_dead start 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.983+0000 7f540d3fe640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f4004800 0x7f53f4004c80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009a90 tx=0x7f53f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49dfe5266cfc4301 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.983+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8039070 con 0x7f53f4004800 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.983+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f802c680 con 0x7f53f4004800 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.983+0000 7f540cbfd640 1 -- 172.21.15.46:0/3476027877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f8009690 con 0x7f53f4004800 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 -- 172.21.15.46:0/3476027877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f4004800 msgr2=0x7f53f4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f4004800 0x7f53f4004c80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f53f8009a90 tx=0x7f53f4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 -- 172.21.15.46:0/3476027877 shutdown_connections 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 --2- 172.21.15.46:0/3476027877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53f4004800 0x7f53f4004c80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 -- 172.21.15.46:0/3476027877 >> 172.21.15.46:0/3476027877 conn(0x7f5408067b40 msgr2=0x7f5408067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 -- 172.21.15.46:0/3476027877 shutdown_connections 2024-04-15T15:17:20.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.984+0000 7f540fe63640 1 -- 172.21.15.46:0/3476027877 wait complete. 2024-04-15T15:17:20.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.985+0000 7f540fe63640 1 Processor -- start 2024-04-15T15:17:20.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.985+0000 7f540fe63640 1 -- start start 2024-04-15T15:17:20.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540fe63640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f4004800 0x7f54081f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540fe63640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54080cb6e0 0x7f54080cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540fe63640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 0x7f54080cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540fe63640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54080d1a90 con 0x7f54080cd7a0 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540fe63640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f54080d45b0 con 0x7f54080cb6e0 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540fe63640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f54080d1c10 con 0x7f53f4004800 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 0x7f54080cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 0x7f54080cdc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60592/0 (socket says 172.21.15.46:60592) 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 -- 172.21.15.46:0/1391233947 learned_addr learned my addr 172.21.15.46:0/1391233947 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540dbff640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f4004800 0x7f54081f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540d3fe640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54080cb6e0 0x7f54080cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 -- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f4004800 msgr2=0x7f54081f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f4004800 0x7f54081f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 -- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54080cb6e0 msgr2=0x7f54080cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54080cb6e0 0x7f54080cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540e400640 1 -- 172.21.15.46:0/1391233947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5408147950 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.986+0000 7f540d3fe640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54080cb6e0 0x7f54080cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.987+0000 7f540e400640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 0x7f54080cdc20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f54080cb690 tx=0x7f540000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.987+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400019070 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.987+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5400003b80 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.987+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5408146b90 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.987+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54000050d0 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.987+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5408147150 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.988+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54080689e0 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.988+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f540001c070 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.989+0000 7f53fe7fc640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f53ec06c980 0x7f53ec06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.989+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f540008b2d0 con 0x7f54080cd7a0 2024-04-15T15:17:20.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.989+0000 7f540dbff640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f53ec06c980 0x7f53ec06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.989+0000 7f540dbff640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f53ec06c980 0x7f53ec06ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5404001eb0 tx=0x7f5404007040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:20.993+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5400058f10 con 0x7f54080cd7a0 2024-04-15T15:17:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.178+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5408161350 con 0x7f54080cd7a0 2024-04-15T15:17:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.179+0000 7f53fe7fc640 1 -- 172.21.15.46:0/1391233947 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+16943 (secure 0 0 0) 0x7f540005d210 con 0x7f54080cd7a0 2024-04-15T15:17:21.181 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:21.181 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":41,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:20.720625+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:21.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f53ec06c980 msgr2=0x7f53ec06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f53ec06c980 0x7f53ec06ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5404001eb0 tx=0x7f5404007040 comp rx=0 tx=0).stop 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 msgr2=0x7f54080cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 0x7f54080cdc20 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f54080cb690 tx=0x7f540000e960 comp rx=0 tx=0).stop 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 shutdown_connections 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f53ec06c980 0x7f53ec06ee40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54080cd7a0 0x7f54080cdc20 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54080cb6e0 0x7f54080cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 --2- 172.21.15.46:0/1391233947 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f53f4004800 0x7f54081f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 >> 172.21.15.46:0/1391233947 conn(0x7f5408067b40 msgr2=0x7f54080b77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 shutdown_connections 2024-04-15T15:17:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.181+0000 7f540fe63640 1 -- 172.21.15.46:0/1391233947 wait complete. 2024-04-15T15:17:21.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:21.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:21.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.301+0000 7f5c8958d640 1 Processor -- start 2024-04-15T15:17:21.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.301+0000 7f5c8958d640 1 -- start start 2024-04-15T15:17:21.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c8958d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c84072d80 0x7f5c84071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c8958d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c840d0da0 0x7f5c840af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c8958d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c8415db40 con 0x7f5c840af3e0 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c8958d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5c8415cfc0 con 0x7f5c84072d80 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c8958d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8415d2a0 con 0x7f5c840d0da0 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c82ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c84072d80 0x7f5c84071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c827fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c840af3e0 0x7f5c840d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55880/0 (socket says 172.21.15.46:55880) 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c827fc640 1 -- 172.21.15.46:0/3424118631 learned_addr learned my addr 172.21.15.46:0/3424118631 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c837fe640 1 --2- 172.21.15.46:0/3424118631 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c840d0da0 0x7f5c840af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3424118631 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c840d0da0 msgr2=0x7f5c840af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3424118631 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c840d0da0 0x7f5c840af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3424118631 >> v1:172.21.15.46:6789/0 conn(0x7f5c840af3e0 legacy=0x7f5c840d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c81ffb640 1 -- 172.21.15.46:0/3424118631 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1197364917 0 0) 0x7f5c8415db40 con 0x7f5c840af3e0 2024-04-15T15:17:21.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.302+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3424118631 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8415d580 con 0x7f5c84072d80 2024-04-15T15:17:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.303+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3424118631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84072d80 0x7f5c84071d10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009d90 tx=0x7f5c7402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11bec25f3e9b61f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.303+0000 7f5c81ffb640 1 -- 172.21.15.46:0/3424118631 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74039070 con 0x7f5c84072d80 2024-04-15T15:17:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.303+0000 7f5c81ffb640 1 -- 172.21.15.46:0/3424118631 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7402cbc0 con 0x7f5c84072d80 2024-04-15T15:17:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.303+0000 7f5c81ffb640 1 -- 172.21.15.46:0/3424118631 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74034440 con 0x7f5c84072d80 2024-04-15T15:17:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 -- 172.21.15.46:0/3424118631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84072d80 msgr2=0x7f5c84071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 --2- 172.21.15.46:0/3424118631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84072d80 0x7f5c84071d10 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009d90 tx=0x7f5c7402be10 comp rx=0 tx=0).stop 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 -- 172.21.15.46:0/3424118631 shutdown_connections 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 --2- 172.21.15.46:0/3424118631 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c840d0da0 0x7f5c840af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 --2- 172.21.15.46:0/3424118631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84072d80 0x7f5c84071d10 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 -- 172.21.15.46:0/3424118631 >> 172.21.15.46:0/3424118631 conn(0x7f5c8406be00 msgr2=0x7f5c8406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 -- 172.21.15.46:0/3424118631 shutdown_connections 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.304+0000 7f5c8958d640 1 -- 172.21.15.46:0/3424118631 wait complete. 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.305+0000 7f5c8958d640 1 Processor -- start 2024-04-15T15:17:21.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.305+0000 7f5c8958d640 1 -- start start 2024-04-15T15:17:21.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c8958d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c84072d80 0x7f5c840cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c8958d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840af3e0 0x7f5c8414d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c8958d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 0x7f5c8414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c8958d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8415cd10 con 0x7f5c840d0da0 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c8958d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8415cb90 con 0x7f5c840af3e0 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c8958d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8406e060 con 0x7f5c84072d80 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 0x7f5c8414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 0x7f5c8414d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60606/0 (socket says 172.21.15.46:60606) 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 -- 172.21.15.46:0/212842579 learned_addr learned my addr 172.21.15.46:0/212842579 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 -- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c84072d80 msgr2=0x7f5c840cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c827fc640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840af3e0 0x7f5c8414d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c84072d80 0x7f5c840cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 -- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840af3e0 msgr2=0x7f5c8414d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840af3e0 0x7f5c8414d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.306+0000 7f5c837fe640 1 -- 172.21.15.46:0/212842579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c84159d80 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.307+0000 7f5c837fe640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 0x7f5c8414d7f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00d5a0 tx=0x7f5c7c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.307+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0125f0 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.307+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7c019070 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.307+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0129f0 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.307+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c8415a060 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.307+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c840b3400 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.308+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c8415a580 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.308+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5c7c012b90 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.309+0000 7f5c7b7fe640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c6406c980 0x7f5c6406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.309+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f5c7c08c100 con 0x7f5c840d0da0 2024-04-15T15:17:21.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.309+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c6406c980 0x7f5c6406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.309+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c6406c980 0x7f5c6406ee40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5c74036000 tx=0x7f5c7402e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.313+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c8415a580 con 0x7f5c840d0da0 2024-04-15T15:17:21.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.497+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5c840689e0 con 0x7f5c840d0da0 2024-04-15T15:17:21.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.497+0000 7f5c7b7fe640 1 -- 172.21.15.46:0/212842579 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f5c7c05a3a0 con 0x7f5c840d0da0 2024-04-15T15:17:21.499 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c6406c980 msgr2=0x7f5c6406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c6406c980 0x7f5c6406ee40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5c74036000 tx=0x7f5c7402e000 comp rx=0 tx=0).stop 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 msgr2=0x7f5c8414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 0x7f5c8414d7f0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00d5a0 tx=0x7f5c7c00da60 comp rx=0 tx=0).stop 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 shutdown_connections 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c6406c980 0x7f5c6406ee40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840d0da0 0x7f5c8414d7f0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840af3e0 0x7f5c8414d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 --2- 172.21.15.46:0/212842579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c84072d80 0x7f5c840cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 >> 172.21.15.46:0/212842579 conn(0x7f5c8406be00 msgr2=0x7f5c84071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 shutdown_connections 2024-04-15T15:17:21.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.499+0000 7f5c8958d640 1 -- 172.21.15.46:0/212842579 wait complete. 2024-04-15T15:17:21.512 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.618+0000 7fe2810c1640 1 Processor -- start 2024-04-15T15:17:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.618+0000 7fe2810c1640 1 -- start start 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe2810c1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe27c15e9a0 0x7fe27c0d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe2810c1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0d1400 0x7fe27c0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe2810c1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe27c0d1e60 con 0x7fe27c0cf750 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe2810c1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe27c0d18a0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe2810c1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe27c0d1b80 con 0x7fe27c0d1400 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe27ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe27c15e9a0 0x7fe27c0d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe27ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe27c15e9a0 0x7fe27c0d0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60612/0 (socket says 172.21.15.46:60612) 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe27ad9d640 1 -- 172.21.15.46:0/2303793759 learned_addr learned my addr 172.21.15.46:0/2303793759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.619+0000 7fe27a59c640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0d1400 0x7fe27c0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.620+0000 7fe27ad9d640 1 -- 172.21.15.46:0/2303793759 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0d1400 msgr2=0x7fe27c0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.620+0000 7fe27ad9d640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0d1400 0x7fe27c0cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.620+0000 7fe27ad9d640 1 -- 172.21.15.46:0/2303793759 >> v1:172.21.15.78:6789/0 conn(0x7fe27c0cf750 legacy=0x7fe27c0d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.620+0000 7fe27ad9d640 1 -- 172.21.15.46:0/2303793759 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c06cff0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.620+0000 7fe27a59c640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0d1400 0x7fe27c0cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.620+0000 7fe27ad9d640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0d0ec0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009e50 tx=0x7fe26c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa6659fcc1caf8e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe279d9b640 1 -- 172.21.15.46:0/2303793759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c02cb00 con 0x7fe27c15e9a0 2024-04-15T15:17:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe279d9b640 1 -- 172.21.15.46:0/2303793759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe26c02cca0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe279d9b640 1 -- 172.21.15.46:0/2303793759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c034610 con 0x7fe27c15e9a0 2024-04-15T15:17:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 -- 172.21.15.46:0/2303793759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 msgr2=0x7fe27c0d0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0d0ec0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009e50 tx=0x7fe26c02bf60 comp rx=0 tx=0).stop 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 -- 172.21.15.46:0/2303793759 shutdown_connections 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0d1400 0x7fe27c0cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 --2- 172.21.15.46:0/2303793759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0d0ec0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 -- 172.21.15.46:0/2303793759 >> 172.21.15.46:0/2303793759 conn(0x7fe27c09d950 msgr2=0x7fe27c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 -- 172.21.15.46:0/2303793759 shutdown_connections 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.621+0000 7fe2810c1640 1 -- 172.21.15.46:0/2303793759 wait complete. 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 Processor -- start 2024-04-15T15:17:21.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 -- start start 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0cf750 0x7fe27c153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe27c0d1400 0x7fe27c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0b9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c0d24e0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe27c0d2360 con 0x7fe27c0d1400 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe2810c1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe27c0d32b0 con 0x7fe27c0cf750 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.623+0000 7fe27b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0b9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0b9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60624/0 (socket says 172.21.15.46:60624) 2024-04-15T15:17:21.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 -- 172.21.15.46:0/1040009767 learned_addr learned my addr 172.21.15.46:0/1040009767 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27ad9d640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0cf750 0x7fe27c153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27a59c640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe27c0d1400 0x7fe27c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 -- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0cf750 msgr2=0x7fe27c153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0cf750 0x7fe27c153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 -- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe27c0d1400 msgr2=0x7fe27c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe27c0d1400 0x7fe27c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 -- 172.21.15.46:0/1040009767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27c154040 con 0x7fe27c15e9a0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe27b59e640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0b9610 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c3d0 tx=0x7fe27400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274017070 con 0x7fe27c15e9a0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2740052d0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe27c0c82d0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe274003830 con 0x7fe27c15e9a0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.624+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe27c0c8890 con 0x7fe27c15e9a0 2024-04-15T15:17:21.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.625+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe27c0690f0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.626+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe27401a070 con 0x7fe27c15e9a0 2024-04-15T15:17:21.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.626+0000 7fe2737fe640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe25c06c980 0x7fe25c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.626+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe27408aff0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.626+0000 7fe27ad9d640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe25c06c980 0x7fe25c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.627+0000 7fe27ad9d640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe25c06c980 0x7fe25c06ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe26c035000 tx=0x7fe26c030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.631+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe27c154040 con 0x7fe27c15e9a0 2024-04-15T15:17:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.815+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe27c06cd30 con 0x7fe27c15e9a0 2024-04-15T15:17:21.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.815+0000 7fe2737fe640 1 -- 172.21.15.46:0/1040009767 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7fe274058cf0 con 0x7fe27c15e9a0 2024-04-15T15:17:21.817 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe25c06c980 msgr2=0x7fe25c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe25c06c980 0x7fe25c06ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe26c035000 tx=0x7fe26c030000 comp rx=0 tx=0).stop 2024-04-15T15:17:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 msgr2=0x7fe27c0b9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0b9610 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe27400c3d0 tx=0x7fe27400c890 comp rx=0 tx=0).stop 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 shutdown_connections 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe25c06c980 0x7fe25c06ee40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27c15e9a0 0x7fe27c0b9610 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe27c0d1400 0x7fe27c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 --2- 172.21.15.46:0/1040009767 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe27c0cf750 0x7fe27c153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 >> 172.21.15.46:0/1040009767 conn(0x7fe27c09d950 msgr2=0x7fe27c06e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 shutdown_connections 2024-04-15T15:17:21.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.817+0000 7fe2810c1640 1 -- 172.21.15.46:0/1040009767 wait complete. 2024-04-15T15:17:21.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:21.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.936+0000 7f6cb4014640 1 Processor -- start 2024-04-15T15:17:21.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.936+0000 7f6cb4014640 1 -- start start 2024-04-15T15:17:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb4014640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0b3160 0x7f6cac15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb4014640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cac15cb50 con 0x7f6cac0d0690 2024-04-15T15:17:21.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb4014640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cac15bfd0 con 0x7f6cac0ae280 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb4014640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6cac15c2b0 con 0x7f6cac0b3160 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb1db0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6cac0d0690 0x7f6cac0d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55882/0 (socket says 172.21.15.46:55882) 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb1db0640 1 -- 172.21.15.46:0/2537573482 learned_addr learned my addr 172.21.15.46:0/2537573482 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb15af640 1 --2- 172.21.15.46:0/2537573482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0b3160 0x7f6cac15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2378272006 0 0) 0x7f6cac15cb50 con 0x7f6cac0d0690 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c98003520 con 0x7f6cac0d0690 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553801185 0 0) 0x7f6cac15bfd0 con 0x7f6cac0ae280 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cac15cb50 con 0x7f6cac0ae280 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4075071108 0 0) 0x7f6c98003520 con 0x7f6cac0d0690 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb15af640 1 -- 172.21.15.46:0/2537573482 >> v1:172.21.15.78:6789/0 conn(0x7f6cac0ae280 legacy=0x7f6cac0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb15af640 1 -- 172.21.15.46:0/2537573482 >> v1:172.21.15.46:6789/0 conn(0x7f6cac0d0690 legacy=0x7f6cac0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.937+0000 7f6cb15af640 1 -- 172.21.15.46:0/2537573482 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac15c590 con 0x7f6cac0b3160 2024-04-15T15:17:21.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.938+0000 7f6cb15af640 1 --2- 172.21.15.46:0/2537573482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0b3160 0x7f6cac15b690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0097d0 tx=0x7f6c9c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2277f851dac022b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.938+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c037070 con 0x7f6cac0b3160 2024-04-15T15:17:21.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.938+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9c030510 con 0x7f6cac0b3160 2024-04-15T15:17:21.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.938+0000 7f6cb0dae640 1 -- 172.21.15.46:0/2537573482 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8001670 con 0x7f6cac0b3160 2024-04-15T15:17:21.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 -- 172.21.15.46:0/2537573482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0b3160 msgr2=0x7f6cac15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 --2- 172.21.15.46:0/2537573482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0b3160 0x7f6cac15b690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c0097d0 tx=0x7f6c9c02b800 comp rx=0 tx=0).stop 2024-04-15T15:17:21.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 -- 172.21.15.46:0/2537573482 shutdown_connections 2024-04-15T15:17:21.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 --2- 172.21.15.46:0/2537573482 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0b3160 0x7f6cac15b690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 -- 172.21.15.46:0/2537573482 >> 172.21.15.46:0/2537573482 conn(0x7f6cac06be00 msgr2=0x7f6cac06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 -- 172.21.15.46:0/2537573482 shutdown_connections 2024-04-15T15:17:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.939+0000 7f6cb4014640 1 -- 172.21.15.46:0/2537573482 wait complete. 2024-04-15T15:17:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.940+0000 7f6cb4014640 1 Processor -- start 2024-04-15T15:17:21.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.940+0000 7f6cb4014640 1 -- start start 2024-04-15T15:17:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb4014640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 0x7f6cac149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb4014640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cac0b3160 0x7f6cac14cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb4014640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0d0690 0x7f6cac0b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb4014640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac15bbd0 con 0x7f6cac0ae280 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb4014640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac0b4fb0 con 0x7f6cac0b3160 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb4014640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6cac15bd50 con 0x7f6cac0d0690 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 0x7f6cac149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 0x7f6cac149a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60632/0 (socket says 172.21.15.46:60632) 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 -- 172.21.15.46:0/228279783 learned_addr learned my addr 172.21.15.46:0/228279783 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb25b1640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0d0690 0x7f6cac0b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb15af640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cac0b3160 0x7f6cac14cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 -- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0d0690 msgr2=0x7f6cac0b1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0d0690 0x7f6cac0b1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 -- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cac0b3160 msgr2=0x7f6cac14cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cac0b3160 0x7f6cac14cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 -- 172.21.15.46:0/228279783 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac09a610 con 0x7f6cac0ae280 2024-04-15T15:17:21.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.941+0000 7f6cb1db0640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 0x7f6cac149a30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c750 tx=0x7f6ca800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.942+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8005ad0 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.942+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca8005c70 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.942+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac09a8f0 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.942+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8005e60 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.942+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac09aeb0 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.943+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cac0689e0 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.943+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6ca80072b0 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.944+0000 7f6ca27fc640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6c9006c980 0x7f6c9006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.944+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f6ca808bba0 con 0x7f6cac0ae280 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.944+0000 7f6cb15af640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6c9006c980 0x7f6c9006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.944+0000 7f6cb15af640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6c9006c980 0x7f6c9006ee40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c034000 tx=0x7f6c9c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:21.948+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ca8019300 con 0x7f6cac0ae280 2024-04-15T15:17:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.132+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6cac0cfec0 con 0x7f6cac0ae280 2024-04-15T15:17:22.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.132+0000 7f6ca27fc640 1 -- 172.21.15.46:0/228279783 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7f6ca8059f00 con 0x7f6cac0ae280 2024-04-15T15:17:22.134 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6c9006c980 msgr2=0x7f6c9006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6c9006c980 0x7f6c9006ee40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c034000 tx=0x7f6c9c008030 comp rx=0 tx=0).stop 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 msgr2=0x7f6cac149a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 0x7f6cac149a30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c750 tx=0x7f6ca800cc10 comp rx=0 tx=0).stop 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 shutdown_connections 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6c9006c980 0x7f6c9006ee40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6cac0d0690 0x7f6cac0b1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cac0b3160 0x7f6cac14cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 --2- 172.21.15.46:0/228279783 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cac0ae280 0x7f6cac149a30 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.134+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 >> 172.21.15.46:0/228279783 conn(0x7f6cac06be00 msgr2=0x7f6cac074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.135+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 shutdown_connections 2024-04-15T15:17:22.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.135+0000 7f6cb4014640 1 -- 172.21.15.46:0/228279783 wait complete. 2024-04-15T15:17:22.147 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.252+0000 7fe73ea4a640 1 Processor -- start 2024-04-15T15:17:22.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.252+0000 7fe73ea4a640 1 -- start start 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.252+0000 7fe73ea4a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fe73815e700 0x7fe7380b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe73ea4a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7380b47a0 con 0x7fe7380d3270 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe73ea4a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7380b4a80 con 0x7fe7380d2360 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe73ea4a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fe7380b53b0 con 0x7fe73815e700 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe73cfe7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe7380d3270 0x7fe7380cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55890/0 (socket says 172.21.15.46:55890) 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe73cfe7640 1 -- 172.21.15.46:0/4109313442 learned_addr learned my addr 172.21.15.46:0/4109313442 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe737fff640 1 --2- 172.21.15.46:0/4109313442 >> v2:172.21.15.78:3300/0 conn(0x7fe73815e700 0x7fe7380b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe736ffd640 1 -- 172.21.15.46:0/4109313442 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 93518147 0 0) 0x7fe7380b47a0 con 0x7fe7380d3270 2024-04-15T15:17:22.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe736ffd640 1 -- 172.21.15.46:0/4109313442 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe720003520 con 0x7fe7380d3270 2024-04-15T15:17:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe737fff640 1 -- 172.21.15.46:0/4109313442 >> v1:172.21.15.78:6789/0 conn(0x7fe7380d2360 legacy=0x7fe7380d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe737fff640 1 -- 172.21.15.46:0/4109313442 >> v1:172.21.15.46:6789/0 conn(0x7fe7380d3270 legacy=0x7fe7380cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.253+0000 7fe737fff640 1 -- 172.21.15.46:0/4109313442 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7380b4d60 con 0x7fe73815e700 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.254+0000 7fe737fff640 1 --2- 172.21.15.46:0/4109313442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73815e700 0x7fe7380b5840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe7380b5530 tx=0x7fe72802b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f9cf4b674988255 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.254+0000 7fe736ffd640 1 -- 172.21.15.46:0/4109313442 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728035020 con 0x7fe73815e700 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.254+0000 7fe736ffd640 1 -- 172.21.15.46:0/4109313442 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe728031890 con 0x7fe73815e700 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.254+0000 7fe736ffd640 1 -- 172.21.15.46:0/4109313442 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728031c40 con 0x7fe73815e700 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 -- 172.21.15.46:0/4109313442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73815e700 msgr2=0x7fe7380b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/4109313442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73815e700 0x7fe7380b5840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe7380b5530 tx=0x7fe72802b930 comp rx=0 tx=0).stop 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 -- 172.21.15.46:0/4109313442 shutdown_connections 2024-04-15T15:17:22.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/4109313442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe73815e700 0x7fe7380b5840 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 -- 172.21.15.46:0/4109313442 >> 172.21.15.46:0/4109313442 conn(0x7fe73808bf60 msgr2=0x7fe7380d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 -- 172.21.15.46:0/4109313442 shutdown_connections 2024-04-15T15:17:22.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.255+0000 7fe73ea4a640 1 -- 172.21.15.46:0/4109313442 wait complete. 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.256+0000 7fe73ea4a640 1 Processor -- start 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.256+0000 7fe73ea4a640 1 -- start start 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73ea4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7380d2360 0x7fe7381519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73ea4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 0x7fe73815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73ea4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe73815e700 0x7fe7380cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73ea4a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7380b5d80 con 0x7fe7380d3270 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73ea4a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe7380b4450 con 0x7fe7380d2360 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73ea4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe7380d1d40 con 0x7fe73815e700 2024-04-15T15:17:22.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 0x7fe73815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 0x7fe73815ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60648/0 (socket says 172.21.15.46:60648) 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 -- 172.21.15.46:0/687988688 learned_addr learned my addr 172.21.15.46:0/687988688 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe737fff640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7380d2360 0x7fe7381519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73cfe7640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe73815e700 0x7fe7380cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 -- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe73815e700 msgr2=0x7fe7380cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe73815e700 0x7fe7380cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 -- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7380d2360 msgr2=0x7fe7381519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7380d2360 0x7fe7381519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 -- 172.21.15.46:0/687988688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7380ce4a0 con 0x7fe7380d3270 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe737fff640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7380d2360 0x7fe7381519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe73cfe7640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe73815e700 0x7fe7380cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.257+0000 7fe7377fe640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 0x7fe73815ec30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00c480 tx=0x7fe72c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.258+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72c0052d0 con 0x7fe7380d3270 2024-04-15T15:17:22.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.258+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe72c005470 con 0x7fe7380d3270 2024-04-15T15:17:22.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.258+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7381612c0 con 0x7fe7380d3270 2024-04-15T15:17:22.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.258+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72c012870 con 0x7fe7380d3270 2024-04-15T15:17:22.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.258+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe738161880 con 0x7fe7380d3270 2024-04-15T15:17:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.259+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7380689e0 con 0x7fe7380d3270 2024-04-15T15:17:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.259+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe72c012a10 con 0x7fe7380d3270 2024-04-15T15:17:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.260+0000 7fe734ff9640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe72406c9b0 0x7fe72406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.260+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe72c08b980 con 0x7fe7380d3270 2024-04-15T15:17:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.260+0000 7fe737fff640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe72406c9b0 0x7fe72406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.260+0000 7fe737fff640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe72406c9b0 0x7fe72406ee70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe728036000 tx=0x7fe728008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.264+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe72c059ca0 con 0x7fe7380d3270 2024-04-15T15:17:22.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.462+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe7380ba500 con 0x7fe7380d3270 2024-04-15T15:17:22.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.462+0000 7fe734ff9640 1 -- 172.21.15.46:0/687988688 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) ==== 80+0+0 (secure 0 0 0) 0x7fe72c0595a0 con 0x7fe7380d3270 2024-04-15T15:17:22.464 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2024-04-15T15:17:22.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe72406c9b0 msgr2=0x7fe72406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe72406c9b0 0x7fe72406ee70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe728036000 tx=0x7fe728008030 comp rx=0 tx=0).stop 2024-04-15T15:17:22.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 msgr2=0x7fe73815ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 0x7fe73815ec30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe72c00c480 tx=0x7fe72c00c940 comp rx=0 tx=0).stop 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 shutdown_connections 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe72406c9b0 0x7fe72406ee70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe73815e700 0x7fe7380cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7380d3270 0x7fe73815ec30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 --2- 172.21.15.46:0/687988688 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7380d2360 0x7fe7381519f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 >> 172.21.15.46:0/687988688 conn(0x7fe73808bf60 msgr2=0x7fe7380d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 shutdown_connections 2024-04-15T15:17:22.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.464+0000 7fe73ea4a640 1 -- 172.21.15.46:0/687988688 wait complete. 2024-04-15T15:17:22.477 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-04-15T15:17:22.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:22.477 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:22.510 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[44755]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:22.512 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:22.602 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:17:22.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:22.602 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:17:22.622 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:22.622 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:22.638 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15004 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:22.638 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi046.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-04-15T15:17:22.639 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:17:22.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:22.639 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:17:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:22.694 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-15T15:17:22.694 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:22.694 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:22.712 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056355,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16861 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:22.713 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi078.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-04-15T15:17:22.713 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:17:22.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:22.713 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:17:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:22.771 INFO:teuthology.orchestra.run.smithi078.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-15T15:17:22.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:17:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.875+0000 7f9e9a50e640 1 Processor -- start 2024-04-15T15:17:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.875+0000 7f9e9a50e640 1 -- start start 2024-04-15T15:17:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e9a50e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9409b720 0x7f9e9409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e9a50e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9406d0a0 con 0x7f9e9415ea00 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e9a50e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e940d0a60 con 0x7f9e9409af10 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e9a50e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9409b370 con 0x7f9e9409b720 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e93fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9e9415ea00 0x7f9e940d1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55900/0 (socket says 172.21.15.46:55900) 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e93fff640 1 -- 172.21.15.46:0/386472448 learned_addr learned my addr 172.21.15.46:0/386472448 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e937fe640 1 --2- 172.21.15.46:0/386472448 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9409b720 0x7f9e9409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415155681 0 0) 0x7f9e9406d0a0 con 0x7f9e9415ea00 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e7c003520 con 0x7f9e9415ea00 2024-04-15T15:17:22.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1454076328 0 0) 0x7f9e940d0a60 con 0x7f9e9409af10 2024-04-15T15:17:22.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e9406d0a0 con 0x7f9e9409af10 2024-04-15T15:17:22.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3298090894 0 0) 0x7f9e7c003520 con 0x7f9e9415ea00 2024-04-15T15:17:22.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e937fe640 1 -- 172.21.15.46:0/386472448 >> v1:172.21.15.78:6789/0 conn(0x7f9e9409af10 legacy=0x7f9e940cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e937fe640 1 -- 172.21.15.46:0/386472448 >> v1:172.21.15.46:6789/0 conn(0x7f9e9415ea00 legacy=0x7f9e940d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.876+0000 7f9e937fe640 1 -- 172.21.15.46:0/386472448 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e940d0d40 con 0x7f9e9409b720 2024-04-15T15:17:22.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.877+0000 7f9e937fe640 1 --2- 172.21.15.46:0/386472448 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9409b720 0x7f9e9409a9d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9e880097d0 tx=0x7f9e8802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e2cd7d1ab67394f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.877+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88035020 con 0x7f9e9409b720 2024-04-15T15:17:22.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.877+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e88031800 con 0x7f9e9409b720 2024-04-15T15:17:22.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.877+0000 7f9e92ffd640 1 -- 172.21.15.46:0/386472448 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88031bb0 con 0x7f9e9409b720 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 -- 172.21.15.46:0/386472448 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9409b720 msgr2=0x7f9e9409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/386472448 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9409b720 0x7f9e9409a9d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9e880097d0 tx=0x7f9e8802b8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 -- 172.21.15.46:0/386472448 shutdown_connections 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/386472448 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9409b720 0x7f9e9409a9d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 -- 172.21.15.46:0/386472448 >> 172.21.15.46:0/386472448 conn(0x7f9e940b3e00 msgr2=0x7f9e940b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 -- 172.21.15.46:0/386472448 shutdown_connections 2024-04-15T15:17:22.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.878+0000 7f9e9a50e640 1 -- 172.21.15.46:0/386472448 wait complete. 2024-04-15T15:17:22.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.879+0000 7f9e9a50e640 1 Processor -- start 2024-04-15T15:17:22.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.879+0000 7f9e9a50e640 1 -- start start 2024-04-15T15:17:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e9a50e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 0x7f9e94156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e9a50e640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e9409b720 0x7f9e941f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e9a50e640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9415ea00 0x7f9e941f8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e9a50e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9406cd00 con 0x7f9e9409af10 2024-04-15T15:17:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e9a50e640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f9e940d10b0 con 0x7f9e9409b720 2024-04-15T15:17:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e9a50e640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9406ce80 con 0x7f9e9415ea00 2024-04-15T15:17:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e93fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 0x7f9e94156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e93fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 0x7f9e94156390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60650/0 (socket says 172.21.15.46:60650) 2024-04-15T15:17:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e93fff640 1 -- 172.21.15.46:0/647049656 learned_addr learned my addr 172.21.15.46:0/647049656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e937fe640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e9409b720 0x7f9e941f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e93fff640 1 -- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9415ea00 msgr2=0x7f9e941f8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e98aab640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9415ea00 0x7f9e941f8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e93fff640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9415ea00 0x7f9e941f8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.880+0000 7f9e93fff640 1 -- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e9409b720 msgr2=0x7f9e941f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e93fff640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e9409b720 0x7f9e941f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e93fff640 1 -- 172.21.15.46:0/647049656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e940c6a10 con 0x7f9e9409af10 2024-04-15T15:17:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e937fe640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e9409b720 0x7f9e941f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e93fff640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 0x7f9e94156390 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400c390 tx=0x7f9e8400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84017070 con 0x7f9e9409af10 2024-04-15T15:17:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e84005ce0 con 0x7f9e9409af10 2024-04-15T15:17:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e940c6cf0 con 0x7f9e9409af10 2024-04-15T15:17:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84007310 con 0x7f9e9409af10 2024-04-15T15:17:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.881+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e940c6630 con 0x7f9e9409af10 2024-04-15T15:17:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.882+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e940689e0 con 0x7f9e9409af10 2024-04-15T15:17:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.883+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9e8401a070 con 0x7f9e9409af10 2024-04-15T15:17:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.883+0000 7f9e90ff9640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9e8006c980 0x7f9e8006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.883+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f9e8408b170 con 0x7f9e9409af10 2024-04-15T15:17:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.883+0000 7f9e937fe640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9e8006c980 0x7f9e8006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.884+0000 7f9e937fe640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9e8006c980 0x7f9e8006ee40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9e88036000 tx=0x7f9e88008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:22.887+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e84059400 con 0x7f9e9409af10 2024-04-15T15:17:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.114+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9e940b38c0 con 0x7f9e9409af10 2024-04-15T15:17:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.114+0000 7f9e90ff9640 1 -- 172.21.15.46:0/647049656 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f9e84058d00 con 0x7f9e9409af10 2024-04-15T15:17:23.116 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9e8006c980 msgr2=0x7f9e8006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9e8006c980 0x7f9e8006ee40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9e88036000 tx=0x7f9e88008030 comp rx=0 tx=0).stop 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 msgr2=0x7f9e94156390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 0x7f9e94156390 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400c390 tx=0x7f9e8400c850 comp rx=0 tx=0).stop 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 shutdown_connections 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f9e8006c980 0x7f9e8006ee40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f9e9415ea00 0x7f9e941f8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e9409b720 0x7f9e941f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 --2- 172.21.15.46:0/647049656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e9409af10 0x7f9e94156390 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.116+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 >> 172.21.15.46:0/647049656 conn(0x7f9e940b3e00 msgr2=0x7f9e940af170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.117+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 shutdown_connections 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23.117+0000 7f9e9a50e640 1 -- 172.21.15.46:0/647049656 wait complete. 2024-04-15T15:17:23.129 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:17:23.129 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:17:23.129 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-15T15:17:23.129 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:17:23.129 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:23.130 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:17:23.130 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda 2024-04-15T15:17:23.130 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:17:23.130 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:17:23.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.130 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-15T15:17:23.146 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:17:23.146 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:17:23.146 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:17:23.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.146 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:17:23.146 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42609sec preferred_lft 42609sec 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:23.147 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 12:e9:35:22:9f:8f brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::50d2:bff:fe30:917d/64 scope link 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 1e:a1:9f:32:6c:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::1ca1:9fff:fe32:6cb0/64 scope link 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 12:e9:35:22:9f:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:23.148 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::10e9:35ff:fe22:9f8f/64 scope link 2024-04-15T15:17:23.149 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:23.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.149 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:17:23.214 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-15T15:17:23.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.215 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-15T15:17:23.272 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-15T15:17:23.272 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:17:23.272 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:17:23.272 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:17:23.272 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:17:23.272 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:17:23.273 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:17:23.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:17:23.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:17:23.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:17:23.273 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:17:23.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:17:23.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:23.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.411 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:17:23.427 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:17:23.427 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:17:23.427 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:17:23.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-04-15T15:17:23.495 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:17:23.496 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:23.496 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:23.551 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:17:23.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:23.552 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:23.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:17:23.675 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:17:23.732 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:23.733 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:23.734 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:23.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.735 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:17:23.789 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:17:23.789 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda) 2024-04-15T15:17:23.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:17:23.859 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:17:23.917 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:17:23.918 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:23.919 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:23.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:23.920 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:17:23.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2024-04-15T15:17:23.919+0000 7f5ae8567480 -1 init, newargv = 0x55b2de58d6d0 newargc=15 2024-04-15T15:17:23.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[55604]: starting ceph client 2024-04-15T15:17:23.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[55604]: starting fuse 2024-04-15T15:17:23.936 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:17:23.937 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:17:23.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:17:23.937 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-04-15T15:17:23.937 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-04-15T15:17:23.937 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-04-15T15:17:23.937 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-04-15T15:17:23.937 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:17:23.938 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2024-04-15T15:17:23.939 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-04-15T15:17:24.062 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.55604.asok 2024-04-15T15:17:24.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:24.073 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55604.asok 2024-04-15T15:17:24.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:24.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55604.asok status 2024-04-15T15:17:24.212 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:24.212 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-04-15T15:17:24.212 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "55604", 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda" 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5856, 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-04-15T15:17:24.213 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5856 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 661709174 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 661709174 2024-04-15T15:17:24.214 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5856 192.168.0.1:0/661709174", 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/661709174", 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 5, 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 17, 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 41, 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-04-15T15:17:24.215 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:24.221 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:17:24.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.324+0000 7f5c89e47640 1 Processor -- start 2024-04-15T15:17:24.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.324+0000 7f5c89e47640 1 -- start start 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c89e47640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c840d3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c89e47640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c840cfda0 0x7f5c840d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c89e47640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c8415ecf0 con 0x7f5c840b6110 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c89e47640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5c840d21d0 con 0x7f5c840cfda0 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c89e47640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5c840d24b0 con 0x7f5c8415e910 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c840cfda0 0x7f5c840d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c840cfda0 0x7f5c840d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60658/0 (socket says 172.21.15.46:60658) 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3049085326 learned_addr learned my addr 172.21.15.46:0/3049085326 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c837fe640 1 --2- 172.21.15.46:0/3049085326 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c840d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3049085326 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 msgr2=0x7f5c840d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:24.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3049085326 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c840d3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3049085326 >> v1:172.21.15.78:6789/0 conn(0x7f5c840b6110 legacy=0x7f5c8406cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.325+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3049085326 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c840d2790 con 0x7f5c840cfda0 2024-04-15T15:17:24.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.326+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3049085326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c840d0200 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009870 tx=0x7f5c7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93ae898b010526b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.326+0000 7f5c827fc640 1 -- 172.21.15.46:0/3049085326 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78034070 con 0x7f5c840cfda0 2024-04-15T15:17:24.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.326+0000 7f5c827fc640 1 -- 172.21.15.46:0/3049085326 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c78030840 con 0x7f5c840cfda0 2024-04-15T15:17:24.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.326+0000 7f5c827fc640 1 -- 172.21.15.46:0/3049085326 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78030c10 con 0x7f5c840cfda0 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 -- 172.21.15.46:0/3049085326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 msgr2=0x7f5c840d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3049085326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c840d0200 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009870 tx=0x7f5c7802b980 comp rx=0 tx=0).stop 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 -- 172.21.15.46:0/3049085326 shutdown_connections 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3049085326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c840d0200 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3049085326 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c840d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 -- 172.21.15.46:0/3049085326 >> 172.21.15.46:0/3049085326 conn(0x7f5c8408bf60 msgr2=0x7f5c840d4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 -- 172.21.15.46:0/3049085326 shutdown_connections 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.327+0000 7f5c89e47640 1 -- 172.21.15.46:0/3049085326 wait complete. 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.328+0000 7f5c89e47640 1 Processor -- start 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.328+0000 7f5c89e47640 1 -- start start 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c89e47640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840b6110 0x7f5c84158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c89e47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c8415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c89e47640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c8415c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c89e47640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5c840d37b0 con 0x7f5c840cfda0 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c89e47640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5c840d2d50 con 0x7f5c840b6110 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c89e47640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5c840d5470 con 0x7f5c8415e910 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c8415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c8415b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60672/0 (socket says 172.21.15.46:60672) 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3819756769 learned_addr learned my addr 172.21.15.46:0/3819756769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c837fe640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840b6110 0x7f5c84158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c83fff640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c8415c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 msgr2=0x7f5c8415c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c8415c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840b6110 msgr2=0x7f5c84158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840b6110 0x7f5c84158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 -- 172.21.15.46:0/3819756769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c840ce350 con 0x7f5c840cfda0 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c837fe640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840b6110 0x7f5c84158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.329+0000 7f5c82ffd640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c8415b200 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5c7802b9c0 tx=0x7f5c78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.330+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78048070 con 0x7f5c840cfda0 2024-04-15T15:17:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.330+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c78030640 con 0x7f5c840cfda0 2024-04-15T15:17:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.330+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78034040 con 0x7f5c840cfda0 2024-04-15T15:17:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.330+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c841f4dc0 con 0x7f5c840cfda0 2024-04-15T15:17:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.330+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c8415aa00 con 0x7f5c840cfda0 2024-04-15T15:17:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.331+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c840689e0 con 0x7f5c840cfda0 2024-04-15T15:17:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.331+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5c7804b070 con 0x7f5c840cfda0 2024-04-15T15:17:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.332+0000 7f5c6bfff640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c7006c980 0x7f5c7006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.332+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f5c780add60 con 0x7f5c840cfda0 2024-04-15T15:17:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.332+0000 7f5c837fe640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c7006c980 0x7f5c7006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.333+0000 7f5c837fe640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c7006c980 0x7f5c7006ee40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c840a2570 tx=0x7f5c74008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.336+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c78030810 con 0x7f5c840cfda0 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.562+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5c840b6e60 con 0x7f5c840cfda0 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.563+0000 7f5c6bfff640 1 -- 172.21.15.46:0/3819756769 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f5c7807b790 con 0x7f5c840cfda0 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c7006c980 msgr2=0x7f5c7006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c7006c980 0x7f5c7006ee40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5c840a2570 tx=0x7f5c74008040 comp rx=0 tx=0).stop 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 msgr2=0x7f5c8415b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c8415b200 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5c7802b9c0 tx=0x7f5c78004730 comp rx=0 tx=0).stop 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 shutdown_connections 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5c7006c980 0x7f5c7006ee40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5c8415e910 0x7f5c8415c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c840cfda0 0x7f5c8415b200 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 --2- 172.21.15.46:0/3819756769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c840b6110 0x7f5c84158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 >> 172.21.15.46:0/3819756769 conn(0x7f5c8408bf60 msgr2=0x7f5c8415bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 shutdown_connections 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:24.565+0000 7f5c89e47640 1 -- 172.21.15.46:0/3819756769 wait complete. 2024-04-15T15:17:24.577 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:17:24.577 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:17:24.577 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-04-15T15:17:24.578 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:17:24.578 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:24.578 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:17:24.578 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda 2024-04-15T15:17:24.578 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:17:24.578 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:17:24.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:24.578 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:24.599 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42594sec preferred_lft 42594sec 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 2e:a8:38:23:6a:89 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.600 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::60de:b7ff:fe5c:109c/64 scope link 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 9e:e5:17:fe:02:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::9ce5:17ff:fefe:2aa/64 scope link 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 2e:a8:38:23:6a:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::2ca8:38ff:fe23:6a89/64 scope link 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:24.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:24.602 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:17:24.672 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-04-15T15:17:24.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:24.674 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-04-15T15:17:24.734 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-04-15T15:17:24.735 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:17:24.735 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:17:24.735 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:17:24.735 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:17:24.735 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:17:24.735 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:17:24.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:17:24.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:17:24.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:17:24.735 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:17:24.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:17:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:24.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:24.887 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:17:24.903 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:17:24.903 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:17:24.903 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:17:24.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:24.903 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2024-04-15T15:17:24.972 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:17:24.972 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:24.972 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:25.029 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:17:25.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:25.030 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:25.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:17:25.153 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:25.210 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:17:25.210 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:25.214 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:17:25.215 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:25.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:25.216 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:17:25.266 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:17:25.266 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda) 2024-04-15T15:17:25.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:17:25.337 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:17:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:17:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:17:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:17:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:17:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:25.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:25.397 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:17:25.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:2024-04-15T15:17:25.395+0000 7fe31ec45480 -1 init, newargv = 0x5593addc6010 newargc=15 2024-04-15T15:17:25.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[45245]: starting ceph client 2024-04-15T15:17:25.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[45245]: starting fuse 2024-04-15T15:17:25.413 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:17:25.413 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-15T15:17:25.413 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:17:25.413 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-04-15T15:17:25.413 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-04-15T15:17:25.413 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-04-15T15:17:25.414 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.1")) 2024-04-15T15:17:25.415 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-04-15T15:17:25.535 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.1.45245.asok 2024-04-15T15:17:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:25.546 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45245.asok 2024-04-15T15:17:25.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:25.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45245.asok status 2024-04-15T15:17:25.685 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "1", 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "45245", 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda" 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:25.686 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5569, 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5569 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1416949740 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:25.687 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1416949740 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.5569 192.168.0.1:0/1416949740", 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/1416949740", 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 5, 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 41, 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-04-15T15:17:25.688 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-04-15T15:17:25.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:17:25.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.810+0000 7f4b8624a640 1 Processor -- start 2024-04-15T15:17:25.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.810+0000 7f4b8624a640 1 -- start start 2024-04-15T15:17:25.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.811+0000 7f4b8624a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f4b8009b0e0 0x7f4b800cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.811+0000 7f4b8624a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8006d0a0 con 0x7f4b8015ea00 2024-04-15T15:17:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.811+0000 7f4b8624a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b800d0a60 con 0x7f4b8009b720 2024-04-15T15:17:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b8624a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f4b800d10b0 con 0x7f4b8009b0e0 2024-04-15T15:17:25.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b8015ea00 0x7f4b800d1540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55904/0 (socket says 172.21.15.46:55904) 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7f7fe640 1 -- 172.21.15.46:0/2819364681 learned_addr learned my addr 172.21.15.46:0/2819364681 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7ffff640 1 --2- 172.21.15.46:0/2819364681 >> v2:172.21.15.78:3300/0 conn(0x7f4b8009b0e0 0x7f4b800cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 487873210 0 0) 0x7f4b8006d0a0 con 0x7f4b8015ea00 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b6c003520 con 0x7f4b8015ea00 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427010396 0 0) 0x7f4b800d0a60 con 0x7f4b8009b720 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b8006d0a0 con 0x7f4b8009b720 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7ffff640 1 -- 172.21.15.46:0/2819364681 >> v1:172.21.15.78:6789/0 conn(0x7f4b8009b720 legacy=0x7f4b8009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7ffff640 1 -- 172.21.15.46:0/2819364681 >> v1:172.21.15.46:6789/0 conn(0x7f4b8015ea00 legacy=0x7f4b800d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.812+0000 7f4b7ffff640 1 -- 172.21.15.46:0/2819364681 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b800d0d40 con 0x7f4b8009b0e0 2024-04-15T15:17:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.813+0000 7f4b7ffff640 1 --2- 172.21.15.46:0/2819364681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b0e0 0x7f4b800cc960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009d90 tx=0x7f4b7802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f973a28a479dad8c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.813+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78039070 con 0x7f4b8009b0e0 2024-04-15T15:17:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.813+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b7802cc60 con 0x7f4b8009b0e0 2024-04-15T15:17:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.813+0000 7f4b7effd640 1 -- 172.21.15.46:0/2819364681 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78034660 con 0x7f4b8009b0e0 2024-04-15T15:17:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 -- 172.21.15.46:0/2819364681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b0e0 msgr2=0x7f4b800cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 --2- 172.21.15.46:0/2819364681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b0e0 0x7f4b800cc960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009d90 tx=0x7f4b7802beb0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 -- 172.21.15.46:0/2819364681 shutdown_connections 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 --2- 172.21.15.46:0/2819364681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b0e0 0x7f4b800cc960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 -- 172.21.15.46:0/2819364681 >> 172.21.15.46:0/2819364681 conn(0x7f4b800b3e00 msgr2=0x7f4b800b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 -- 172.21.15.46:0/2819364681 shutdown_connections 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.814+0000 7f4b8624a640 1 -- 172.21.15.46:0/2819364681 wait complete. 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.815+0000 7f4b8624a640 1 Processor -- start 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.815+0000 7f4b8624a640 1 -- start start 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b8624a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 0x7f4b801527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b8624a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b720 0x7f4b80153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b8624a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b8015ea00 0x7f4b80155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b8624a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8006ce80 con 0x7f4b8009b0e0 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b8624a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8006cd00 con 0x7f4b8009b720 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b8624a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4b800b1590 con 0x7f4b8015ea00 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 0x7f4b801527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 0x7f4b801527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60674/0 (socket says 172.21.15.46:60674) 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 -- 172.21.15.46:0/1896367268 learned_addr learned my addr 172.21.15.46:0/1896367268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7ffff640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b8015ea00 0x7f4b80155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b77fff640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b720 0x7f4b80153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 -- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b8015ea00 msgr2=0x7f4b80155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b8015ea00 0x7f4b80155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 -- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b720 msgr2=0x7f4b80153f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b720 0x7f4b80153f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 -- 172.21.15.46:0/1896367268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8015be90 con 0x7f4b8009b0e0 2024-04-15T15:17:25.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7f7fe640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 0x7f4b801527b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000c410 tx=0x7f4b7000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.816+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70017070 con 0x7f4b8009b0e0 2024-04-15T15:17:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.817+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8015b310 con 0x7f4b8009b0e0 2024-04-15T15:17:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.817+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b70005ce0 con 0x7f4b8009b0e0 2024-04-15T15:17:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.817+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b700072f0 con 0x7f4b8009b0e0 2024-04-15T15:17:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.817+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b8015b8d0 con 0x7f4b8009b0e0 2024-04-15T15:17:25.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.818+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b800689e0 con 0x7f4b8009b0e0 2024-04-15T15:17:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.818+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4b7001a070 con 0x7f4b8009b0e0 2024-04-15T15:17:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.818+0000 7f4b7cff9640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4b6006c980 0x7f4b6006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.819+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f4b7008b0d0 con 0x7f4b8009b0e0 2024-04-15T15:17:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.819+0000 7f4b77fff640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4b6006c980 0x7f4b6006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.819+0000 7f4b77fff640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4b6006c980 0x7f4b6006ee40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4b800c9c50 tx=0x7f4b68009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:25.823+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b70058de0 con 0x7f4b8009b0e0 2024-04-15T15:17:26.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.047+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4b8015e020 con 0x7f4b8009b0e0 2024-04-15T15:17:26.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.047+0000 7f4b7cff9640 1 -- 172.21.15.46:0/1896367268 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f4b7005d0e0 con 0x7f4b8009b0e0 2024-04-15T15:17:26.049 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4b6006c980 msgr2=0x7f4b6006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4b6006c980 0x7f4b6006ee40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4b800c9c50 tx=0x7f4b68009040 comp rx=0 tx=0).stop 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 msgr2=0x7f4b801527b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 0x7f4b801527b0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000c410 tx=0x7f4b7000c8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 shutdown_connections 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4b6006c980 0x7f4b6006ee40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4b8015ea00 0x7f4b80155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b8009b720 0x7f4b80153f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 --2- 172.21.15.46:0/1896367268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b8009b0e0 0x7f4b801527b0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 >> 172.21.15.46:0/1896367268 conn(0x7f4b800b3e00 msgr2=0x7f4b8006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 shutdown_connections 2024-04-15T15:17:26.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26.049+0000 7f4b8624a640 1 -- 172.21.15.46:0/1896367268 wait complete. 2024-04-15T15:17:26.061 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:17:26.062 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:17:26.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.063 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-15T15:17:26.080 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:17:26.080 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:17:26.080 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:17:26.080 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.080 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-15T15:17:26.080 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.081 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:26.081 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:26.081 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-15T15:17:26.081 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:26.081 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:26.081 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-15T15:17:26.082 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:17:26.082 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:26.082 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-15T15:17:26.082 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-15T15:17:26.082 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42606sec preferred_lft 42606sec 2024-04-15T15:17:26.083 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-15T15:17:26.083 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.083 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:26.083 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:26.083 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-15T15:17:26.083 INFO:teuthology.orchestra.run.smithi046.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:17:26.084 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 12:e9:35:22:9f:8f brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:26.084 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:17:26.084 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.084 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::50d2:bff:fe30:917d/64 scope link 2024-04-15T15:17:26.084 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.084 INFO:teuthology.orchestra.run.smithi046.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:26.085 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 1e:a1:9f:32:6c:b0 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:26.085 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::1ca1:9fff:fe32:6cb0/64 scope link 2024-04-15T15:17:26.085 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.085 INFO:teuthology.orchestra.run.smithi046.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:26.085 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 12:e9:35:22:9f:8f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:26.085 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::10e9:35ff:fe22:9f8f/64 scope link 2024-04-15T15:17:26.086 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:26.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.086 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:17:26.146 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-15T15:17:26.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.148 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:17:26.205 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:17:26.205 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-15T15:17:26.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-15T15:17:26.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-15T15:17:26.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-15T15:17:26.206 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-15T15:17:26.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:26.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.345 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:17:26.360 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:17:26.360 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:17:26.360 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:17:26.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.360 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.1 up 2024-04-15T15:17:26.429 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:17:26.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:26.429 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:17:26.485 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:17:26.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:26.486 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:17:26.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:17:26.609 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:26.669 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:17:26.669 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:17:26.669 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:17:26.669 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:17:26.669 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:17:26.670 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:17:26.671 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:26.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.672 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:17:26.725 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:17:26.725 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:17:26.725 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda) 2024-04-15T15:17:26.768 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-04-15T15:17:26.796 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:26.852 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-04-15T15:17:26.853 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:17:26.854 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:26.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:26.855 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-04-15T15:17:26.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:2024-04-15T15:17:26.855+0000 7eff7f341480 -1 init, newargv = 0x560eb5b520b0 newargc=15 2024-04-15T15:17:26.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[56029]: starting ceph client 2024-04-15T15:17:26.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[56029]: starting fuse 2024-04-15T15:17:26.871 INFO:teuthology.orchestra.run.smithi046.stdout:43 2024-04-15T15:17:26.872 INFO:teuthology.orchestra.run.smithi046.stdout:44 2024-04-15T15:17:26.872 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-04-15T15:17:26.872 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-04-15T15:17:26.873 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.2")) 2024-04-15T15:17:26.874 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-04-15T15:17:26.994 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.2.56029.asok 2024-04-15T15:17:26.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:27.004 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.56029.asok 2024-04-15T15:17:27.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:27.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.56029.asok status 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "2", 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "56029", 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda" 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:27.144 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5880, 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5880 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 379974014 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:27.145 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 379974014 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5880 192.168.0.2:0/379974014", 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.2:0/379974014", 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 5, 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 17, 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 41, 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-04-15T15:17:27.146 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:27.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:17:27.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 Processor -- start 2024-04-15T15:17:27.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 -- start start 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88e815c520 0x7f88e8072080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80725c0 0x7f88e80cd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e8072a60 con 0x7f88e80cd570 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f88e80cd9c0 con 0x7f88e815c520 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.255+0000 7f88ed0eb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f88e8072d40 con 0x7f88e80725c0 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e6d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f88e815c520 0x7f88e8072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88dffff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80725c0 0x7f88e80cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e759e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f88e80cd570 0x7f88e8160160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:60272/0 (socket says 172.21.15.46:60272) 2024-04-15T15:17:27.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e759e640 1 -- 172.21.15.46:0/2544035017 learned_addr learned my addr 172.21.15.46:0/2544035017 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:27.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e6d9d640 1 -- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80725c0 msgr2=0x7f88e80cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:27.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e6d9d640 1 --2- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80725c0 0x7f88e80cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e6d9d640 1 -- 172.21.15.46:0/2544035017 >> v1:172.21.15.78:6789/0 conn(0x7f88e80cd570 legacy=0x7f88e8160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88e6d9d640 1 -- 172.21.15.46:0/2544035017 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e8073020 con 0x7f88e815c520 2024-04-15T15:17:27.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.256+0000 7f88dffff640 1 --2- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80725c0 0x7f88e80cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.257+0000 7f88e6d9d640 1 --2- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e815c520 0x7f88e8072080 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88d80097a0 tx=0x7f88d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ce6c0576f22cc47 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.257+0000 7f88e659c640 1 -- 172.21.15.46:0/2544035017 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8035020 con 0x7f88e815c520 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.257+0000 7f88e659c640 1 -- 172.21.15.46:0/2544035017 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d8031830 con 0x7f88e815c520 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.257+0000 7f88e659c640 1 -- 172.21.15.46:0/2544035017 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d8031c00 con 0x7f88e815c520 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 -- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e815c520 msgr2=0x7f88e8072080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e815c520 0x7f88e8072080 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f88d80097a0 tx=0x7f88d802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 -- 172.21.15.46:0/2544035017 shutdown_connections 2024-04-15T15:17:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80725c0 0x7f88e80cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/2544035017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e815c520 0x7f88e8072080 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 -- 172.21.15.46:0/2544035017 >> 172.21.15.46:0/2544035017 conn(0x7f88e8067b10 msgr2=0x7f88e806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 -- 172.21.15.46:0/2544035017 shutdown_connections 2024-04-15T15:17:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.258+0000 7f88ed0eb640 1 -- 172.21.15.46:0/2544035017 wait complete. 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.259+0000 7f88ed0eb640 1 Processor -- start 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.259+0000 7f88ed0eb640 1 -- start start 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88ed0eb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e80725c0 0x7f88e815a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88ed0eb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80cd570 0x7f88e8153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88ed0eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 0x7f88e81f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88ed0eb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f88e815c900 con 0x7f88e815c520 2024-04-15T15:17:27.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88ed0eb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f88e80cf7a0 con 0x7f88e80725c0 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88ed0eb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f88e815ca80 con 0x7f88e80cd570 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 0x7f88e81f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 0x7f88e81f3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60678/0 (socket says 172.21.15.46:60678) 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 -- 172.21.15.46:0/251808192 learned_addr learned my addr 172.21.15.46:0/251808192 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88dffff640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80cd570 0x7f88e8153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e6d9d640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e80725c0 0x7f88e815a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 -- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80cd570 msgr2=0x7f88e8153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80cd570 0x7f88e8153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 -- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e80725c0 msgr2=0x7f88e815a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e80725c0 0x7f88e815a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.260+0000 7f88e759e640 1 -- 172.21.15.46:0/251808192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e80b20c0 con 0x7f88e815c520 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.261+0000 7f88e759e640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 0x7f88e81f3330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f88e000cbc0 tx=0x7f88e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.261+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0017070 con 0x7f88e815c520 2024-04-15T15:17:27.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.261+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e000ad40 con 0x7f88e815c520 2024-04-15T15:17:27.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.261+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e81526e0 con 0x7f88e815c520 2024-04-15T15:17:27.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.261+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0012600 con 0x7f88e815c520 2024-04-15T15:17:27.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.261+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e8159960 con 0x7f88e815c520 2024-04-15T15:17:27.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.262+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e806c970 con 0x7f88e815c520 2024-04-15T15:17:27.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.262+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f88e001a070 con 0x7f88e815c520 2024-04-15T15:17:27.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.263+0000 7f88df7fe640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f88c806c9b0 0x7f88c806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.263+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f88e008ba80 con 0x7f88e815c520 2024-04-15T15:17:27.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.263+0000 7f88e6d9d640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f88c806c9b0 0x7f88c806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.264+0000 7f88e6d9d640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f88c806c9b0 0x7f88c806ee70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f88d8036000 tx=0x7f88d803c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.267+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88e008de20 con 0x7f88e815c520 2024-04-15T15:17:27.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.493+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f88e806c440 con 0x7f88e815c520 2024-04-15T15:17:27.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.493+0000 7f88df7fe640 1 -- 172.21.15.46:0/251808192 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f88e80b20c0 con 0x7f88e815c520 2024-04-15T15:17:27.495 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:17:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f88c806c9b0 msgr2=0x7f88c806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f88c806c9b0 0x7f88c806ee70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f88d8036000 tx=0x7f88d803c040 comp rx=0 tx=0).stop 2024-04-15T15:17:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 msgr2=0x7f88e81f3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 0x7f88e81f3330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f88e000cbc0 tx=0x7f88e000a430 comp rx=0 tx=0).stop 2024-04-15T15:17:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 shutdown_connections 2024-04-15T15:17:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f88c806c9b0 0x7f88c806ee70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e815c520 0x7f88e81f3330 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f88e80cd570 0x7f88e8153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 --2- 172.21.15.46:0/251808192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88e80725c0 0x7f88e815a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 >> 172.21.15.46:0/251808192 conn(0x7f88e8067b10 msgr2=0x7f88e815f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 shutdown_connections 2024-04-15T15:17:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:27.495+0000 7f88ed0eb640 1 -- 172.21.15.46:0/251808192 wait complete. 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:17:27.508 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:17:27.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:27.508 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2024-04-15T15:17:27.528 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:17:27.528 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:17:27.528 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:17:27.528 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f0 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: altname enp7s0f1 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 42591sec preferred_lft 42591sec 2024-04-15T15:17:27.529 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::ec4:7aff:fe8f:d4dd/64 scope link 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 2e:a8:38:23:6a:89 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::60de:b7ff:fe5c:109c/64 scope link 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 9e:e5:17:fe:02:aa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::9ce5:17ff:fefe:2aa/64 scope link 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:17:27.530 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 2e:a8:38:23:6a:89 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:27.531 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::2ca8:38ff:fe23:6a89/64 scope link 2024-04-15T15:17:27.531 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:17:27.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:27.531 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:17:27.601 INFO:teuthology.orchestra.run.smithi078.stdout:1 2024-04-15T15:17:27.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:27.603 DEBUG:teuthology.orchestra.run.smithi078:> route 2024-04-15T15:17:27.662 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2024-04-15T15:17:27.662 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:17:27.663 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:17:27.663 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:17:27.663 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:17:27.663 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:17:27.663 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2024-04-15T15:17:27.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:17:27.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:17:27.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:17:27.663 DEBUG:teuthology.orchestra.run.smithi078:> ') 2024-04-15T15:17:27.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:17:27.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:27.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:27.818 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:17:27.834 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:17:27.834 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:17:27.834 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:17:27.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:27.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.1 up 2024-04-15T15:17:27.904 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:17:27.904 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:27.904 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:17:27.960 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:17:27.961 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:17:27.961 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:17:28.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:17:28.085 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:17:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:17:28.143 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:28.144 INFO:teuthology.orchestra.run.smithi078.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:28.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:28.145 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:17:28.198 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:17:28.198 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-04-15T15:17:28.198 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda) 2024-04-15T15:17:28.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2024-04-15T15:17:28.269 DEBUG:teuthology.orchestra.run.smithi078:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/proc 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sys 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/dev 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/security 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/shm 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/pts 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/run 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/cgroup 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/pstore 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/bpf 2024-04-15T15:17:28.326 INFO:teuthology.orchestra.run.smithi078.stdout:/ 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/selinux 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/hugepages 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/dev/mqueue 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/debug 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/tracing 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/fs/fuse/connections 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/sys/kernel/config 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:17:28.327 INFO:teuthology.orchestra.run.smithi078.stdout:/run/user/1000 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:28.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:28.329 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2024-04-15T15:17:28.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:2024-04-15T15:17:28.328+0000 7fefa043a480 -1 init, newargv = 0x55d8af10f2e0 newargc=15 2024-04-15T15:17:28.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:ceph-fuse[45585]: starting ceph client 2024-04-15T15:17:28.337 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:ceph-fuse[45585]: starting fuse 2024-04-15T15:17:28.345 INFO:teuthology.orchestra.run.smithi078.stdout:42 2024-04-15T15:17:28.345 INFO:teuthology.orchestra.run.smithi078.stdout:44 2024-04-15T15:17:28.345 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-04-15T15:17:28.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> import re 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> import os 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:28.346 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> return f 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.3")) 2024-04-15T15:17:28.347 DEBUG:teuthology.orchestra.run.smithi078:> ' 2024-04-15T15:17:28.467 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.3.45585.asok 2024-04-15T15:17:28.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-04-15T15:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:28.477 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.45585.asok 2024-04-15T15:17:28.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:28.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.45585.asok status 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "3", 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "45585", 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda" 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:28.615 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 5222, 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5222 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3524751820 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:28.616 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3524751820 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.5222 192.168.0.2:0/3524751820", 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.2:0/3524751820", 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 5, 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 41, 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2024-04-15T15:17:28.617 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-04-15T15:17:28.623 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'touch file') 2024-04-15T15:17:28.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:17:28.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:17:28.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:28.695 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quiesce.block -v 1 file) 2024-04-15T15:17:28.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 quiesce path /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda --wait 2024-04-15T15:17:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.866+0000 7f3b516d3640 1 Processor -- start 2024-04-15T15:17:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.866+0000 7f3b516d3640 1 -- start start 2024-04-15T15:17:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b516d3640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3b4c0d1670 0x7f3b4c162ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b516d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b4c09b3e0 0x7f3b4c06caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b516d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b4c0d08b0 con 0x7f3b4c1623f0 2024-04-15T15:17:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b516d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3b4c0d1ad0 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b516d3640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3b4c0d0680 con 0x7f3b4c0d1670 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3b4c09b3e0 0x7f3b4c06caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3b4c1623f0 0x7f3b4c1627d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55918/0 (socket says 172.21.15.46:55918) 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4affd640 1 -- 172.21.15.46:0/4144394215 learned_addr learned my addr 172.21.15.46:0/4144394215 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b43fff640 1 --2- 172.21.15.46:0/4144394215 >> v2:172.21.15.78:3300/0 conn(0x7f3b4c0d1670 0x7f3b4c162ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/4144394215 >> v2:172.21.15.78:3300/0 conn(0x7f3b4c0d1670 msgr2=0x7f3b4c162ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/4144394215 >> v2:172.21.15.78:3300/0 conn(0x7f3b4c0d1670 0x7f3b4c162ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/4144394215 >> v1:172.21.15.46:6789/0 conn(0x7f3b4c1623f0 legacy=0x7f3b4c1627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.868+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/4144394215 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4c0d0b90 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.867+0000 7f3b4a7fc640 1 -- 172.21.15.46:0/4144394215 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132111036 0 0) 0x7f3b4c0d08b0 con 0x7f3b4c1623f0 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.868+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/4144394215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c06caf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0c5f80 tx=0x7f3b4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a019bf05f433b980 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.868+0000 7f3b4a7fc640 1 -- 172.21.15.46:0/4144394215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44039070 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.868+0000 7f3b4a7fc640 1 -- 172.21.15.46:0/4144394215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4402cc80 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.868+0000 7f3b4a7fc640 1 -- 172.21.15.46:0/4144394215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44034660 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 -- 172.21.15.46:0/4144394215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 msgr2=0x7f3b4c06caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 --2- 172.21.15.46:0/4144394215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c06caf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0c5f80 tx=0x7f3b4402bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 -- 172.21.15.46:0/4144394215 shutdown_connections 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 --2- 172.21.15.46:0/4144394215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c06caf0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 --2- 172.21.15.46:0/4144394215 >> v2:172.21.15.78:3300/0 conn(0x7f3b4c0d1670 0x7f3b4c162ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 -- 172.21.15.46:0/4144394215 >> 172.21.15.46:0/4144394215 conn(0x7f3b4c0b3f70 msgr2=0x7f3b4c0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 -- 172.21.15.46:0/4144394215 shutdown_connections 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.869+0000 7f3b516d3640 1 -- 172.21.15.46:0/4144394215 wait complete. 2024-04-15T15:17:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.870+0000 7f3b516d3640 1 Processor -- start 2024-04-15T15:17:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.870+0000 7f3b516d3640 1 -- start start 2024-04-15T15:17:28.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b516d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b516d3640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c0d1670 0x7f3b4c160970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b516d3640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3b4c1623f0 0x7f3b4c160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b516d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c09b870 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b516d3640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c0d1150 con 0x7f3b4c0d1670 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b516d3640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3b4c06d030 con 0x7f3b4c1623f0 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c15aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60696/0 (socket says 172.21.15.46:60696) 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 -- 172.21.15.46:0/131088730 learned_addr learned my addr 172.21.15.46:0/131088730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3b4c1623f0 0x7f3b4c160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b43fff640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c0d1670 0x7f3b4c160970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 -- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3b4c1623f0 msgr2=0x7f3b4c160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3b4c1623f0 0x7f3b4c160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 -- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c0d1670 msgr2=0x7f3b4c160970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c0d1670 0x7f3b4c160970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 -- 172.21.15.46:0/131088730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b4c15f7a0 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3b4c1623f0 0x7f3b4c160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b43fff640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c0d1670 0x7f3b4c160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.871+0000 7f3b4affd640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c15aa10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00c430 tx=0x7f3b3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.872+0000 7f3b437fe640 1 -- 172.21.15.46:0/131088730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c0074e0 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.872+0000 7f3b437fe640 1 -- 172.21.15.46:0/131088730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b3c0047e0 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.872+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b4c15fa80 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.872+0000 7f3b437fe640 1 -- 172.21.15.46:0/131088730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c004bd0 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.872+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b4c160040 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.873+0000 7f3b437fe640 1 -- 172.21.15.46:0/131088730 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3b3c011070 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.874+0000 7f3b437fe640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3b2c06c980 0x7f3b2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.874+0000 7f3b437fe640 1 -- 172.21.15.46:0/131088730 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f3b3c08b960 con 0x7f3b4c09b3e0 2024-04-15T15:17:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.874+0000 7f3b43fff640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3b2c06c980 0x7f3b2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.874+0000 7f3b43fff640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3b2c06c980 0x7f3b2c06ee40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c06e4c0 tx=0x7f3b34008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.874+0000 7f3b54396740 1 Processor -- start 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 -- start start 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cc0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cee260 0x55e796cf0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 0x55e796cf1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e796c88ad0 con 0x55e796cc1360 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e796c2c560 con 0x55e796cee260 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55e796bd2c00 con 0x55e796cede80 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 0x55e796cf1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b4b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 0x55e796cf1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60704/0 (socket says 172.21.15.46:60704) 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b4affd640 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cc0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.875+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/526372866 learned_addr learned my addr 172.21.15.46:0/526372866 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:28.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b43fff640 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cee260 0x55e796cf0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 msgr2=0x55e796cc0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cc0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cee260 msgr2=0x55e796cf0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cee260 0x55e796cf0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b4b7fe640 1 -- 172.21.15.46:0/526372866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e796a0b030 con 0x55e796cc1360 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 0x55e796cf1150 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3b4402bf10 tx=0x7f3b4402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8128b0c12860f0f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b40ff9640 1 -- 172.21.15.46:0/526372866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b44039040 con 0x55e796cc1360 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b40ff9640 1 -- 172.21.15.46:0/526372866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4403f440 con 0x55e796cc1360 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.876+0000 7f3b40ff9640 1 -- 172.21.15.46:0/526372866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4403f7f0 con 0x55e796cc1360 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 -- 172.21.15.46:0/526372866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 msgr2=0x55e796cf1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 0x55e796cf1150 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3b4402bf10 tx=0x7f3b4402e000 comp rx=0 tx=0).stop 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 -- 172.21.15.46:0/526372866 shutdown_connections 2024-04-15T15:17:28.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cc1360 0x55e796cf1150 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cee260 0x55e796cf0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 --2- 172.21.15.46:0/526372866 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cc0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 -- 172.21.15.46:0/526372866 >> 172.21.15.46:0/526372866 conn(0x55e796cc3090 msgr2=0x55e796d0bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 -- 172.21.15.46:0/526372866 shutdown_connections 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.877+0000 7f3b54396740 1 -- 172.21.15.46:0/526372866 wait complete. 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 -- start start 2024-04-15T15:17:28.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 Processor -- start 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cf4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 0x55e796d0c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796ce87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e796be6110 con 0x55e796cee260 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55e796c28520 con 0x55e796ce6c20 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55e796cc5730 con 0x55e796cede80 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b54396740 10 client.0 _ll_register_callbacks cb 0x55e79693e410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 0x55e796d0c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 0x55e796d0c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60710/0 (socket says 172.21.15.46:60710) 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.878+0000 7f3b43fff640 1 -- 172.21.15.46:0/1020100095 learned_addr learned my addr 172.21.15.46:0/1020100095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:28.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b4affd640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cf4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796ce87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b43fff640 1 -- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 msgr2=0x55e796cf4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b43fff640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cf4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b43fff640 1 -- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 msgr2=0x55e796ce87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b43fff640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796ce87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b43fff640 1 -- 172.21.15.46:0/1020100095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e79693ec20 con 0x55e796cee260 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b43fff640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 0x55e796d0c190 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c1f7960 tx=0x7f3b340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c1b0ca7376689b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b29ffb640 1 -- 172.21.15.46:0/1020100095 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34025070 con 0x55e796cee260 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b29ffb640 1 -- 172.21.15.46:0/1020100095 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f3b340158e0 con 0x55e796cee260 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b4403f440 con 0x55e796cee260 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b29ffb640 1 -- 172.21.15.46:0/1020100095 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3b3401b040 con 0x55e796cee260 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b54396740 10 client.5901 fetch_fsmap learned FSMap version 17 2024-04-15T15:17:28.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55e7968fe920 con 0x55e796cee260 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.879+0000 7f3b29ffb640 1 -- 172.21.15.46:0/1020100095 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4403f440 con 0x55e796cee260 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b29ffb640 1 -- 172.21.15.46:0/1020100095 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f3b34015ae0 con 0x55e796cee260 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b54396740 10 client.5901 fetch_fsmap finished waiting for FSMap version 17 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b54396740 10 client.5901 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b54396740 20 client.5901 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b54396740 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796ceeea0 0x55e796cef280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b54396740 4 client.5901 mds_command: new command op to 5454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 --> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e796c39f80 con 0x55e796ceeea0 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b4affd640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796ceeea0 0x55e796cef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b4affd640 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796ceeea0 0x55e796cef280 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc6fcdddaed1d81 server_cookie=8b1ab26cdf07b4bd in_seq=0 out_seq=0 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.880+0000 7f3b29ffb640 10 client.5901 ms_handle_connect on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b29ffb640 1 -- 172.21.15.46:0/1020100095 <== mds.0 v2:172.21.15.46:6833/964535021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e796c39f80 con 0x55e796ceeea0 2024-04-15T15:17:28.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b29ffb640 10 client.5901 handle_command_reply: tid=0 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 client.5901 shutdown 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 msgr2=0x55e796d0c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 0x55e796d0c190 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c1f7960 tx=0x7f3b340046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 shutdown_connections 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796ceeea0 0x55e796cef280 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796ce87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cee260 0x55e796d0c190 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b29ffb640 0 client.5901 ms_handle_reset on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 --2- 172.21.15.46:0/1020100095 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cede80 0x55e796cf4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 >> 172.21.15.46:0/1020100095 conn(0x55e796cc1360 msgr2=0x55e796cc4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 shutdown_connections 2024-04-15T15:17:28.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 1 -- 172.21.15.46:0/1020100095 wait complete. 2024-04-15T15:17:28.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.881+0000 7f3b54396740 20 client.5901 trim_cache size 0 max 16384 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.898+0000 7f3b54396740 1 Processor -- start 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.898+0000 7f3b54396740 1 -- start start 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796d7c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b54396740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 0x55e796cef990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796cf2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b54396740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c2013a0 con 0x55e796cede80 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c200380 con 0x55e796ce6c20 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3b4c1f5920 con 0x55e796cee260 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 0x55e796cef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 0x55e796cef990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60724/0 (socket says 172.21.15.46:60724) 2024-04-15T15:17:28.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 -- 172.21.15.46:0/280868994 learned_addr learned my addr 172.21.15.46:0/280868994 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796cf2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b4affd640 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796d7c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 -- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 msgr2=0x55e796cf2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796cf2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 -- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 msgr2=0x55e796d7c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796d7c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 -- 172.21.15.46:0/280868994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e796ce8b10 con 0x55e796cede80 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b4affd640 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796d7c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.899+0000 7f3b43fff640 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 0x55e796cef990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400f950 tx=0x7f3b34022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927e437b750a1602 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b2bfff640 1 -- 172.21.15.46:0/280868994 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34025050 con 0x55e796cede80 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b2bfff640 1 -- 172.21.15.46:0/280868994 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b3401c040 con 0x55e796cede80 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b2bfff640 1 -- 172.21.15.46:0/280868994 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b34015270 con 0x55e796cede80 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 -- 172.21.15.46:0/280868994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 msgr2=0x55e796cef990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 0x55e796cef990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400f950 tx=0x7f3b34022000 comp rx=0 tx=0).stop 2024-04-15T15:17:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 -- 172.21.15.46:0/280868994 shutdown_connections 2024-04-15T15:17:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796cf2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796cede80 0x55e796cef990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 --2- 172.21.15.46:0/280868994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796ce6c20 0x55e796d7c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 -- 172.21.15.46:0/280868994 >> 172.21.15.46:0/280868994 conn(0x55e796cc1360 msgr2=0x55e796d0b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 -- 172.21.15.46:0/280868994 shutdown_connections 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.900+0000 7f3b54396740 1 -- 172.21.15.46:0/280868994 wait complete. 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.901+0000 7f3b54396740 1 -- start start 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.901+0000 7f3b54396740 1 Processor -- start 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.901+0000 7f3b54396740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 0x55e796ce5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.901+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cede80 0x55e796cf4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796d6be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e796cef5b0 con 0x55e796ce6c20 2024-04-15T15:17:28.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4c1ffd20 con 0x55e796cede80 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55e796ce6930 con 0x55e796cee260 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 10 client.0 _ll_register_callbacks cb 0x7f3b3400fe40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 0x55e796ce5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 0x55e796ce5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60736/0 (socket says 172.21.15.46:60736) 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 -- 172.21.15.46:0/4140215857 learned_addr learned my addr 172.21.15.46:0/4140215857 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b43fff640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cede80 0x55e796cf4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4b7fe640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796d6be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 -- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 msgr2=0x55e796d6be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796d6be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 -- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cede80 msgr2=0x55e796cf4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cede80 0x55e796cf4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 -- 172.21.15.46:0/4140215857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e796d2f860 con 0x55e796ce6c20 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b4affd640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 0x55e796ce5b70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c059510 tx=0x7f3b3c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ecac54b49946437 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b29ffb640 1 -- 172.21.15.46:0/4140215857 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c057940 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b29ffb640 1 -- 172.21.15.46:0/4140215857 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f3b3c055bf0 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b3401c040 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b29ffb640 1 -- 172.21.15.46:0/4140215857 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3b3c0589d0 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 10 client.5907 fetch_fsmap learned FSMap version 17 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.902+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x55e796d86120 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b29ffb640 1 -- 172.21.15.46:0/4140215857 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b3401c040 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b29ffb640 1 -- 172.21.15.46:0/4140215857 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7f3b3c058b70 con 0x55e796ce6c20 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b54396740 10 client.5907 fetch_fsmap finished waiting for FSMap version 17 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b54396740 10 client.5907 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b54396740 20 client.5907 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b54396740 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796d6c3d0 0x55e796d1a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b54396740 4 client.5907 mds_command: new command op to 5454 tid=0 multi_id=0 [{"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda", "wait": true}] 2024-04-15T15:17:28.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 --> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] -- command(tid 0: {"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda", "wait": true}) -- 0x55e796ce8310 con 0x55e796d6c3d0 2024-04-15T15:17:28.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b43fff640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796d6c3d0 0x55e796d1a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b43fff640 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796d6c3d0 0x55e796d1a940 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=985162a1ad01d09d server_cookie=10bf9346dfbe7c1 in_seq=0 out_seq=0 2024-04-15T15:17:28.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.903+0000 7f3b29ffb640 10 client.5907 ms_handle_connect on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:28.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b29ffb640 1 -- 172.21.15.46:0/4140215857 <== mds.0 v2:172.21.15.46:6833/964535021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2332 (crc 0 0 0) 0x55e796ce8310 con 0x55e796d6c3d0 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b29ffb640 10 client.5907 handle_command_reply: tid=0 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 client.5907 shutdown 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 msgr2=0x55e796ce5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 0x55e796ce5b70 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c059510 tx=0x7f3b3c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 shutdown_connections 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x55e796d6c3d0 0x55e796d1a940 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55e796cee260 0x55e796d6be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e796cede80 0x55e796cf4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 --2- 172.21.15.46:0/4140215857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e796ce6c20 0x55e796ce5b70 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b29ffb640 0 client.5907 ms_handle_reset on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:28.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.907+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 >> 172.21.15.46:0/4140215857 conn(0x55e796cc1360 msgr2=0x55e796cc7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 shutdown_connections 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b54396740 1 -- 172.21.15.46:0/4140215857 wait complete. 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b54396740 20 client.5907 trim_cache size 0 max 16384 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout: "op": { 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout: "result": 0, 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout: "flag_point": "quiesce complete", 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout: "reqid": { 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": { 2024-04-15T15:17:28.916 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "mds", 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 0 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "tid": 1 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "op_type": "internal_op", 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "internal_op": 5384, 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "op_name": "quiesce_path", 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "events": [ 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904606+0000", 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "initiated" 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.917 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904606+0000", 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "header_read" 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904606+0000", 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "throttled" 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904606+0000", 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "all_read" 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904606+0000", 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "dispatched" 2024-04-15T15:17:28.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904669+0000", 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904689+0000", 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904708+0000", 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.919 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.904729+0000", 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.908441+0000", 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.908452+0000", 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.908461+0000", 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.908471+0000", 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "acquired locks" 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "time": "2024-04-15T15:17:28.908483+0000", 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "event": "quiesce complete" 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "locks": null 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "state": { 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes": 2, 2024-04-15T15:17:28.921 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_quiesced": 1, 2024-04-15T15:17:28.922 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_blocked": 1, 2024-04-15T15:17:28.922 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_dropped": 0, 2024-04-15T15:17:28.922 INFO:teuthology.orchestra.run.smithi046.stdout: "failed": [] 2024-04-15T15:17:28.922 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-15T15:17:28.922 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:28.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3b2c06c980 msgr2=0x7f3b2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3b2c06c980 0x7f3b2c06ee40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c06e4c0 tx=0x7f3b34008040 comp rx=0 tx=0).stop 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 msgr2=0x7f3b4c15aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c15aa10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c00c430 tx=0x7f3b3c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 shutdown_connections 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3b2c06c980 0x7f3b2c06ee40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3b4c1623f0 0x7f3b4c160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3b4c0d1670 0x7f3b4c160970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 --2- 172.21.15.46:0/131088730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3b4c09b3e0 0x7f3b4c15aa10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 >> 172.21.15.46:0/131088730 conn(0x7f3b4c0b3f70 msgr2=0x7f3b4c15c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 shutdown_connections 2024-04-15T15:17:28.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:28.908+0000 7f3b516d3640 1 -- 172.21.15.46:0/131088730 wait complete. 2024-04-15T15:17:28.924 DEBUG:tasks.cephfs.test_quiesce:{'op': {'result': 0, 'flag_point': 'quiesce complete', 'reqid': {'entity': {'type': 'mds', 'num': 0}, 'tid': 1}, 'op_type': 'internal_op', 'internal_op': 5384, 'op_name': 'quiesce_path', 'events': [{'time': '2024-04-15T15:17:28.904606+0000', 'event': 'initiated'}, {'time': '2024-04-15T15:17:28.904606+0000', 'event': 'header_read'}, {'time': '2024-04-15T15:17:28.904606+0000', 'event': 'throttled'}, {'time': '2024-04-15T15:17:28.904606+0000', 'event': 'all_read'}, {'time': '2024-04-15T15:17:28.904606+0000', 'event': 'dispatched'}, {'time': '2024-04-15T15:17:28.904669+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.904689+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.904708+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.904729+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.908441+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.908452+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.908461+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.908471+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:17:28.908483+0000', 'event': 'quiesce complete'}], 'locks': None}, 'state': {'inodes': 2, 'inodes_quiesced': 1, 'inodes_blocked': 1, 'inodes_dropped': 0, 'failed': []}} 2024-04-15T15:17:28.925 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:29.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.028+0000 7fe2b499d640 1 Processor -- start 2024-04-15T15:17:29.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.028+0000 7fe2b499d640 1 -- start start 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2b499d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2b499d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2b009ab40 0x7fe2b009afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2b499d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2b015dbd0 con 0x7fe2b01606b0 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2b499d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fe2b00cd6a0 con 0x7fe2b009ab40 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2b499d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b00cd980 con 0x7fe2b01602d0 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2aed9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2b009ab40 0x7fe2b009afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2add9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2b01606b0 0x7fe2b00d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55932/0 (socket says 172.21.15.46:55932) 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2add9b640 1 -- 172.21.15.46:0/609570217 learned_addr learned my addr 172.21.15.46:0/609570217 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2ae59c640 1 --2- 172.21.15.46:0/609570217 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2aed9d640 1 -- 172.21.15.46:0/609570217 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 msgr2=0x7fe2b00cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2aed9d640 1 --2- 172.21.15.46:0/609570217 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2aed9d640 1 -- 172.21.15.46:0/609570217 >> v1:172.21.15.46:6789/0 conn(0x7fe2b01606b0 legacy=0x7fe2b00d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2aed9d640 1 -- 172.21.15.46:0/609570217 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b00cdc60 con 0x7fe2b009ab40 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.029+0000 7fe2ad59a640 1 -- 172.21.15.46:0/609570217 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 560522279 0 0) 0x7fe2b015dbd0 con 0x7fe2b01606b0 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.030+0000 7fe2aed9d640 1 --2- 172.21.15.46:0/609570217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b009afd0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00097a0 tx=0x7fe2a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f2b5c40b2e3c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.030+0000 7fe2ad59a640 1 -- 172.21.15.46:0/609570217 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0035020 con 0x7fe2b009ab40 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.030+0000 7fe2ad59a640 1 -- 172.21.15.46:0/609570217 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0031830 con 0x7fe2b009ab40 2024-04-15T15:17:29.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.030+0000 7fe2ad59a640 1 -- 172.21.15.46:0/609570217 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0031c00 con 0x7fe2b009ab40 2024-04-15T15:17:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 -- 172.21.15.46:0/609570217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 msgr2=0x7fe2b009afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 --2- 172.21.15.46:0/609570217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b009afd0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00097a0 tx=0x7fe2a002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 -- 172.21.15.46:0/609570217 shutdown_connections 2024-04-15T15:17:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 --2- 172.21.15.46:0/609570217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b009afd0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 --2- 172.21.15.46:0/609570217 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 -- 172.21.15.46:0/609570217 >> 172.21.15.46:0/609570217 conn(0x7fe2b00d2460 msgr2=0x7fe2b00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 -- 172.21.15.46:0/609570217 shutdown_connections 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.031+0000 7fe2b499d640 1 -- 172.21.15.46:0/609570217 wait complete. 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.032+0000 7fe2b499d640 1 Processor -- start 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.032+0000 7fe2b499d640 1 -- start start 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2b499d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b01592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2b499d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2b499d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b01606b0 0x7fe2b015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2b499d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0160b10 con 0x7fe2b009ab40 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2b499d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b00ce220 con 0x7fe2b01606b0 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2b499d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b009b510 con 0x7fe2b01602d0 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b01592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b01592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60746/0 (socket says 172.21.15.46:60746) 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 -- 172.21.15.46:0/131179445 learned_addr learned my addr 172.21.15.46:0/131179445 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:29.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 -- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 msgr2=0x7fe2b00b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 -- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b01606b0 msgr2=0x7fe2b015b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b01606b0 0x7fe2b015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 -- 172.21.15.46:0/131179445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b0070bd0 con 0x7fe2b009ab40 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.033+0000 7fe2ae59c640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b01592f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800efc0 tx=0x7fe2a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.034+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8018020 con 0x7fe2b009ab40 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.034+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a8014480 con 0x7fe2b009ab40 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.034+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8014850 con 0x7fe2b009ab40 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.034+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b0070eb0 con 0x7fe2b009ab40 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.034+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b00b6080 con 0x7fe2b009ab40 2024-04-15T15:17:29.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.035+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2b00b5cb0 con 0x7fe2b009ab40 2024-04-15T15:17:29.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.035+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe2a8005d10 con 0x7fe2b009ab40 2024-04-15T15:17:29.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.035+0000 7fe2a6ffd640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe29006c9b0 0x7fe29006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.036+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fe2a808c9e0 con 0x7fe2b009ab40 2024-04-15T15:17:29.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.036+0000 7fe2add9b640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe29006c9b0 0x7fe29006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.036+0000 7fe2add9b640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe29006c9b0 0x7fe29006ee70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe298007fd0 tx=0x7fe298007960 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.040+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2a8022070 con 0x7fe2b009ab40 2024-04-15T15:17:29.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.269+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2b00689e0 con 0x7fe2b009ab40 2024-04-15T15:17:29.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.269+0000 7fe2a6ffd640 1 -- 172.21.15.46:0/131179445 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5175 (secure 0 0 0) 0x7fe2a8010040 con 0x7fe2b009ab40 2024-04-15T15:17:29.271 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:29.271 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:03.437143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5454},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5454":{"gid":5454,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/964535021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":964535021},{"type":"v1","addr":"172.21.15.46:6835","nonce":964535021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5454,"qdb_cluster":[5454]},"id":2}]} 2024-04-15T15:17:29.272 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe29006c9b0 msgr2=0x7fe29006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe29006c9b0 0x7fe29006ee70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe298007fd0 tx=0x7fe298007960 comp rx=0 tx=0).stop 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 msgr2=0x7fe2b01592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b01592f0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe2a800efc0 tx=0x7fe2a800c490 comp rx=0 tx=0).stop 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 shutdown_connections 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fe29006c9b0 0x7fe29006ee70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2b01606b0 0x7fe2b015b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fe2b01602d0 0x7fe2b00b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 --2- 172.21.15.46:0/131179445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2b009ab40 0x7fe2b01592f0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 >> 172.21.15.46:0/131179445 conn(0x7fe2b00d2460 msgr2=0x7fe2b00714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 shutdown_connections 2024-04-15T15:17:29.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.272+0000 7fe2b499d640 1 -- 172.21.15.46:0/131179445 wait complete. 2024-04-15T15:17:29.284 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda 0 2024-04-15T15:17:29.404 INFO:teuthology.orchestra.run.smithi046.stdout:[{"ino":1099511627779,"rdev":0,"ctime":"2024-04-15T15:17:28.683031+0000","btime":"2024-04-15T15:17:15.385170+0000","mode":16895,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":6,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-04-15T15:17:28.683031+0000","atime":"2024-04-15T15:17:15.385170+0000","time_warp_seq":0,"change_attr":2,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-04-15T15:17:28.683031+0000","num_files":1,"num_subdirs":0,"change_attr":1},"rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:17:28.711450+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:17:28.711450+0000"},"version":20,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":2,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"waiter":0,"authpin":1},"nref":8,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{"reqid":{"entity":{"type":"mds","num":0},"tid":2}}},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":5222,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5467,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":7},{"client_id":5569,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5856,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":3},{"client_id":5880,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/file","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511629282,"remote_type":"","version":6,"projected_version":6,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-04-15T15:17:29.405 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-04-15T15:17:28.711450+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-04-15T15:17:15.385170+0000", "auth_pins": 1, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-04-15T15:17:15.385170+0000", "change_attr": 2, "client_caps": [ { "client_id": 5222, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5467, "issued": "pAsLsXs", "last_sent": 7, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5569, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5856, "issued": "pAsLsXs", "last_sent": 3, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5880, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-04-15T15:17:28.683031+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511629282, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/file", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 6, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 6, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 1, "mtime": "2024-04-15T15:17:28.683031+0000", "num_files": 1, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1099511627779, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 6, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16895, "mtime": "2024-04-15T15:17:28.683031+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 8, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 2, "request": 1, "waiter": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "lock", "type": "iquiesce", "xlock_by": { "reqid": { "entity": { "num": 0, "type": "mds" }, "tid": 2 } } }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-04-15T15:17:28.711450+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 20, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-04-15T15:17:29.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 ops --flags=locks --path=/tmp/mds.0-ops 2024-04-15T15:17:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.511+0000 7febcc95f640 1 Processor -- start 2024-04-15T15:17:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.511+0000 7febcc95f640 1 -- start start 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febcc95f640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7febc80af340 0x7febc80b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febcc95f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febc80b1620 0x7febc80b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febcc95f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febc80af920 con 0x7febc815ecb0 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febcc95f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7febc815f090 con 0x7febc80b1620 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febcc95f640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7febc80cdf80 con 0x7febc80af340 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febc80b1620 0x7febc80b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7febc815ecb0 0x7febc80b0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55942/0 (socket says 172.21.15.46:55942) 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc659c640 1 -- 172.21.15.46:0/3012371829 learned_addr learned my addr 172.21.15.46:0/3012371829 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febbffff640 1 --2- 172.21.15.46:0/3012371829 >> v2:172.21.15.78:3300/0 conn(0x7febc80af340 0x7febc80b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc6d9d640 1 -- 172.21.15.46:0/3012371829 >> v2:172.21.15.78:3300/0 conn(0x7febc80af340 msgr2=0x7febc80b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3012371829 >> v2:172.21.15.78:3300/0 conn(0x7febc80af340 0x7febc80b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc6d9d640 1 -- 172.21.15.46:0/3012371829 >> v1:172.21.15.46:6789/0 conn(0x7febc815ecb0 legacy=0x7febc80b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc6d9d640 1 -- 172.21.15.46:0/3012371829 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc80afc00 con 0x7febc80b1620 2024-04-15T15:17:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.512+0000 7febc5d9b640 1 -- 172.21.15.46:0/3012371829 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3291406924 0 0) 0x7febc80af920 con 0x7febc815ecb0 2024-04-15T15:17:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.513+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3012371829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b1ab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7febc806a050 tx=0x7febc002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928a00fe7a68e6ec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.513+0000 7febc5d9b640 1 -- 172.21.15.46:0/3012371829 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0039070 con 0x7febc80b1620 2024-04-15T15:17:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.513+0000 7febc5d9b640 1 -- 172.21.15.46:0/3012371829 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc002cd00 con 0x7febc80b1620 2024-04-15T15:17:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.513+0000 7febc5d9b640 1 -- 172.21.15.46:0/3012371829 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0034660 con 0x7febc80b1620 2024-04-15T15:17:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 -- 172.21.15.46:0/3012371829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 msgr2=0x7febc80b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 --2- 172.21.15.46:0/3012371829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b1ab0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7febc806a050 tx=0x7febc002bf50 comp rx=0 tx=0).stop 2024-04-15T15:17:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 -- 172.21.15.46:0/3012371829 shutdown_connections 2024-04-15T15:17:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 --2- 172.21.15.46:0/3012371829 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b1ab0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 --2- 172.21.15.46:0/3012371829 >> v2:172.21.15.78:3300/0 conn(0x7febc80af340 0x7febc80b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 -- 172.21.15.46:0/3012371829 >> 172.21.15.46:0/3012371829 conn(0x7febc80b6870 msgr2=0x7febc80b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 -- 172.21.15.46:0/3012371829 shutdown_connections 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.514+0000 7febcc95f640 1 -- 172.21.15.46:0/3012371829 wait complete. 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.515+0000 7febcc95f640 1 Processor -- start 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.515+0000 7febcc95f640 1 -- start start 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febcc95f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febc80af340 0x7febc8154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febcc95f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febcc95f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7febc815ecb0 0x7febc80b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febcc95f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febc80ce410 con 0x7febc80b1620 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febcc95f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7febc80b4890 con 0x7febc80af340 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febcc95f640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7febc80ce590 con 0x7febc815ecb0 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60758/0 (socket says 172.21.15.46:60758) 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 -- 172.21.15.46:0/3007261473 learned_addr learned my addr 172.21.15.46:0/3007261473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7febc815ecb0 0x7febc80b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febc659c640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febc80af340 0x7febc8154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 -- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7febc815ecb0 msgr2=0x7febc80b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7febc815ecb0 0x7febc80b5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 -- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febc80af340 msgr2=0x7febc8154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febc80af340 0x7febc8154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 -- 172.21.15.46:0/3007261473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc80ca8e0 con 0x7febc80b1620 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbffff640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b5a80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7febb000e9b0 tx=0x7febb000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.516+0000 7febbeffd640 1 -- 172.21.15.46:0/3007261473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0017070 con 0x7febc80b1620 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.517+0000 7febbeffd640 1 -- 172.21.15.46:0/3007261473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb000c9a0 con 0x7febc80b1620 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.517+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc80cabc0 con 0x7febc80b1620 2024-04-15T15:17:29.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.517+0000 7febbeffd640 1 -- 172.21.15.46:0/3007261473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb000cd90 con 0x7febc80b1620 2024-04-15T15:17:29.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.517+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc80cb180 con 0x7febc80b1620 2024-04-15T15:17:29.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.518+0000 7febbeffd640 1 -- 172.21.15.46:0/3007261473 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7febb001a070 con 0x7febc80b1620 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.518+0000 7febbeffd640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feba806c980 0x7feba806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.519+0000 7febbeffd640 1 -- 172.21.15.46:0/3007261473 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7febb008c080 con 0x7febc80b1620 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.519+0000 7febc659c640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feba806c980 0x7feba806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.519+0000 7febc659c640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feba806c980 0x7feba806ee40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7febb8007fd0 tx=0x7febb8007de0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.519+0000 7febcf620740 1 Processor -- start 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 -- start start 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a19eda0 0x56195a19f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 0x56195a1a4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1ce8a0 0x56195a1ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56195a109a80 con 0x56195a1a2b60 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56195a0b0580 con 0x56195a19eda0 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.520+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x56195a105a40 con 0x56195a1ce8a0 2024-04-15T15:17:29.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 0x56195a1a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 0x56195a1a4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60772/0 (socket says 172.21.15.46:60772) 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 -- 172.21.15.46:0/3498699819 learned_addr learned my addr 172.21.15.46:0/3498699819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1ce8a0 0x56195a1ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febc659c640 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a19eda0 0x56195a19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 -- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1ce8a0 msgr2=0x56195a1ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1ce8a0 0x56195a1ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 -- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a19eda0 msgr2=0x56195a19f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a19eda0 0x56195a19f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 -- 172.21.15.46:0/3498699819 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195a0c4ad0 con 0x56195a1a2b60 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7febbffff640 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 0x56195a1a4600 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7febb0056120 tx=0x7febb0059e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed776135bb443fae server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7feba7fff640 1 -- 172.21.15.46:0/3498699819 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0021040 con 0x56195a1a2b60 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7feba7fff640 1 -- 172.21.15.46:0/3498699819 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb00721d0 con 0x56195a1a2b60 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.521+0000 7feba7fff640 1 -- 172.21.15.46:0/3498699819 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0059110 con 0x56195a1a2b60 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 -- 172.21.15.46:0/3498699819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 msgr2=0x56195a1a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 0x56195a1a4600 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7febb0056120 tx=0x7febb0059e00 comp rx=0 tx=0).stop 2024-04-15T15:17:29.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 -- 172.21.15.46:0/3498699819 shutdown_connections 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1ce8a0 0x56195a1ced20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1a2b60 0x56195a1a4600 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 --2- 172.21.15.46:0/3498699819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a19eda0 0x56195a19f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 -- 172.21.15.46:0/3498699819 >> 172.21.15.46:0/3498699819 conn(0x56195a1a0e20 msgr2=0x56195a1a2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 -- 172.21.15.46:0/3498699819 shutdown_connections 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 -- 172.21.15.46:0/3498699819 wait complete. 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.522+0000 7febcf620740 1 -- start start 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 Processor -- start 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1e92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a1d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1d1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56195a109a80 con 0x56195a1d0e90 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x56195a0c3940 con 0x56195a1ce8a0 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x56195a1a2ff0 con 0x56195a1c9d30 2024-04-15T15:17:29.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febcf620740 10 client.0 _ll_register_callbacks cb 0x56195a0c3350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.523+0000 7febc6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1d1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60784/0 (socket says 172.21.15.46:60784) 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc6d9d640 1 -- 172.21.15.46:0/2087391758 learned_addr learned my addr 172.21.15.46:0/2087391758 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1e92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febbffff640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a1d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 -- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 msgr2=0x56195a1d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a1d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 -- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 msgr2=0x56195a1d1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1d1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 -- 172.21.15.46:0/2087391758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561959e1c420 con 0x56195a1ce8a0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febbffff640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a1d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc6d9d640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febc659c640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1e92c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7febc806f100 tx=0x7febb8000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a608476c3cadbb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7feba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7feba57fa640 1 -- 172.21.15.46:0/2087391758 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8020580 con 0x56195a1ce8a0 2024-04-15T15:17:29.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7feba57fa640 1 -- 172.21.15.46:0/2087391758 <== mon.1 v2:172.21.15.78:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7febb8020720 con 0x56195a1ce8a0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.524+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56195a1174a0 con 0x56195a1ce8a0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7feba57fa640 1 -- 172.21.15.46:0/2087391758 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7febb802e050 con 0x56195a1ce8a0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 10 client.5623 fetch_fsmap learned FSMap version 17 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x561959ee8830 con 0x56195a1ce8a0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7feba57fa640 1 -- 172.21.15.46:0/2087391758 <== mon.1 v2:172.21.15.78:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56195a1174a0 con 0x56195a1ce8a0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7feba57fa640 1 -- 172.21.15.46:0/2087391758 <== mon.1 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7febb8031020 con 0x56195a1ce8a0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 10 client.5623 fetch_fsmap finished waiting for FSMap version 17 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 10 client.5623 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 20 client.5623 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1c8eb0 0x56195a206ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 4 client.5623 mds_command: new command op to 5454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 --> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56195a111310 con 0x56195a1c8eb0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febbffff640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1c8eb0 0x56195a206ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7febbffff640 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1c8eb0 0x56195a206ef0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbd9aaf4af6a9e4b server_cookie=3e77eef96df8563 in_seq=0 out_seq=0 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.525+0000 7feba57fa640 10 client.5623 ms_handle_connect on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7feba57fa640 1 -- 172.21.15.46:0/2087391758 <== mds.0 v2:172.21.15.46:6833/964535021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56195a111310 con 0x56195a1c8eb0 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7feba57fa640 10 client.5623 handle_command_reply: tid=0 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 client.5623 shutdown 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 msgr2=0x56195a1e92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1e92c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7febc806f100 tx=0x7febb8000ed0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 shutdown_connections 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1c8eb0 0x56195a206ef0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a1d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7feba57fa640 0 client.5623 ms_handle_reset on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 --2- 172.21.15.46:0/2087391758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1e92c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 >> 172.21.15.46:0/2087391758 conn(0x56195a19eda0 msgr2=0x56195a1eadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 shutdown_connections 2024-04-15T15:17:29.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 1 -- 172.21.15.46:0/2087391758 wait complete. 2024-04-15T15:17:29.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.526+0000 7febcf620740 20 client.5623 trim_cache size 0 max 16384 2024-04-15T15:17:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.542+0000 7febcf620740 1 Processor -- start 2024-04-15T15:17:29.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.542+0000 7febcf620740 1 -- start start 2024-04-15T15:17:29.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febcf620740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 0x56195a1c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1d0e90 0x56195a1e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febcf620740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56195a1a0540 con 0x56195a1c9d30 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7febc8201330 con 0x56195a1ce8a0 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7febc8200370 con 0x56195a1d0e90 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 0x56195a1c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 0x56195a1c7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60794/0 (socket says 172.21.15.46:60794) 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 -- 172.21.15.46:0/2828697459 learned_addr learned my addr 172.21.15.46:0/2828697459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc6d9d640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1d0e90 0x56195a1e8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febbffff640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 -- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1d0e90 msgr2=0x56195a1e8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1d0e90 0x56195a1e8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 -- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 msgr2=0x56195a1cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc659c640 1 -- 172.21.15.46:0/2828697459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56195a1e3cb0 con 0x56195a1c9d30 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febbffff640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.543+0000 7febc6d9d640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1d0e90 0x56195a1e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febc659c640 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 0x56195a1c7a40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7febb8007470 tx=0x7febb8021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d335c8ff93733b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7feba77fe640 1 -- 172.21.15.46:0/2828697459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8020580 con 0x56195a1c9d30 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7feba77fe640 1 -- 172.21.15.46:0/2828697459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb8026070 con 0x56195a1c9d30 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7feba77fe640 1 -- 172.21.15.46:0/2828697459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb8020970 con 0x56195a1c9d30 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 -- 172.21.15.46:0/2828697459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 msgr2=0x56195a1c7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 0x56195a1c7a40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7febb8007470 tx=0x7febb8021000 comp rx=0 tx=0).stop 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 -- 172.21.15.46:0/2828697459 shutdown_connections 2024-04-15T15:17:29.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1d0e90 0x56195a1e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a1cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 --2- 172.21.15.46:0/2828697459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1c9d30 0x56195a1c7a40 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 -- 172.21.15.46:0/2828697459 >> 172.21.15.46:0/2828697459 conn(0x56195a19eda0 msgr2=0x56195a199ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 -- 172.21.15.46:0/2828697459 shutdown_connections 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.544+0000 7febcf620740 1 -- 172.21.15.46:0/2828697459 wait complete. 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.545+0000 7febcf620740 1 -- start start 2024-04-15T15:17:29.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.545+0000 7febcf620740 1 Processor -- start 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7febc81601c0 con 0x56195a1d0e90 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7febc815fb70 con 0x56195a1ce8a0 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x56195a1cefd0 con 0x56195a1c9d30 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febcf620740 10 client.0 _ll_register_callbacks cb 0x7febb8004da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:29.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1cd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45112/0 (socket says 172.21.15.46:45112) 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 -- 172.21.15.46:0/3362716648 learned_addr learned my addr 172.21.15.46:0/3362716648 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc659c640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febbffff640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 -- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 msgr2=0x56195a19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 -- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 msgr2=0x56195a19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a19e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 -- 172.21.15.46:0/3362716648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56195a266c90 con 0x56195a1d0e90 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7febc6d9d640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1cd680 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7febc0009e50 tx=0x7febc002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac909509de49553 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.546+0000 7feba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:29.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7feba57fa640 1 -- 172.21.15.46:0/3362716648 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0050060 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7feba57fa640 1 -- 172.21.15.46:0/3362716648 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7febc0034440 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febb8026070 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7feba57fa640 1 -- 172.21.15.46:0/3362716648 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7febc0039040 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 10 client.5937 fetch_fsmap learned FSMap version 17 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x56195a1ced10 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7feba57fa640 1 -- 172.21.15.46:0/3362716648 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febb8026070 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7feba57fa640 1 -- 172.21.15.46:0/3362716648 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 3494+0+0 (secure 0 0 0) 0x7febc0059080 con 0x56195a1d0e90 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 10 client.5937 fetch_fsmap finished waiting for FSMap version 17 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 10 client.5937 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 20 client.5937 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1cdbc0 0x56195a1cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:29.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 4 client.5937 mds_command: new command op to 5454 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}] 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 --> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}) -- 0x56195a08d8b0 con 0x56195a1cdbc0 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febc659c640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1cdbc0 0x56195a1cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7febc659c640 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1cdbc0 0x56195a1cdfd0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7477f5b70d6b6b9c server_cookie=85508582bd9ab383 in_seq=0 out_seq=0 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.547+0000 7feba57fa640 10 client.5937 ms_handle_connect on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7feba57fa640 1 -- 172.21.15.46:0/3362716648 <== mds.0 v2:172.21.15.46:6833/964535021 1 ==== command_reply(tid 0: 0 ) ==== 8+0+33 (crc 0 0 0) 0x56195a08d8b0 con 0x56195a1cdbc0 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7feba57fa640 10 client.5937 handle_command_reply: tid=0 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 client.5937 shutdown 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 msgr2=0x56195a1cd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1cd680 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7febc0009e50 tx=0x7febc002e070 comp rx=0 tx=0).stop 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 shutdown_connections 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:6833/964535021,v1:172.21.15.46:6835/964535021] conn(0x56195a1cdbc0 0x56195a1cdfd0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56195a1d0e90 0x56195a1cd680 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56195a1ce8a0 0x56195a19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 --2- 172.21.15.46:0/3362716648 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x56195a1c9d30 0x56195a19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7feba57fa640 0 client.5937 ms_handle_reset on v2:172.21.15.46:6833/964535021 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 >> 172.21.15.46:0/3362716648 conn(0x56195a19eda0 msgr2=0x56195a1cb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 shutdown_connections 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.548+0000 7febcf620740 1 -- 172.21.15.46:0/3362716648 wait complete. 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcf620740 20 client.5937 trim_cache size 0 max 16384 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/tmp/mds.0-ops" 2024-04-15T15:17:29.557 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feba806c980 msgr2=0x7feba806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feba806c980 0x7feba806ee40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7febb8007fd0 tx=0x7febb8007de0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 msgr2=0x7febc80b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b5a80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7febb000e9b0 tx=0x7febb000ee70 comp rx=0 tx=0).stop 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 shutdown_connections 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feba806c980 0x7feba806ee40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7febc815ecb0 0x7febc80b5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc80b1620 0x7febc80b5a80 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 --2- 172.21.15.46:0/3007261473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7febc80af340 0x7febc8154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 >> 172.21.15.46:0/3007261473 conn(0x7febc80b6870 msgr2=0x7febc806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 shutdown_connections 2024-04-15T15:17:29.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:29.549+0000 7febcc95f640 1 -- 172.21.15.46:0/3007261473 wait complete. 2024-04-15T15:17:29.562 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-04-15T15:17:29.577 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.odAHecdOLb 2024-04-15T15:17:29.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /tmp/mds.0-ops /tmp/tmp.odAHecdOLb 2024-04-15T15:17:29.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.odAHecdOLb 2024-04-15T15:17:29.773 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.odAHecdOLb is 8KB 2024-04-15T15:17:29.794 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.odAHecdOLb 2024-04-15T15:17:29.809 DEBUG:tasks.cephfs.filesystem:read 7765B of ops 2024-04-15T15:17:29.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda --path=/tmp/mds.0-cache 2024-04-15T15:17:29.968 INFO:teuthology.orchestra.run.smithi046.stdout:{"path":"/tmp/mds.0-cache"} 2024-04-15T15:17:29.969 DEBUG:tasks.cephfs.filesystem:_json_asok output { "path": "/tmp/mds.0-cache" } 2024-04-15T15:17:29.969 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-04-15T15:17:29.985 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.2TrYWYRD4T 2024-04-15T15:17:29.985 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /tmp/mds.0-cache /tmp/tmp.2TrYWYRD4T 2024-04-15T15:17:30.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.2TrYWYRD4T 2024-04-15T15:17:30.181 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.2TrYWYRD4T is 6KB 2024-04-15T15:17:30.230 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.2TrYWYRD4T 2024-04-15T15:17:30.246 DEBUG:tasks.cephfs.filesystem:read 6360B of cache 2024-04-15T15:17:30.246 INFO:tasks.cephfs.test_quiesce:qp = 1; qi = 1; qib = 1; qina = 0 2024-04-15T15:17:30.247 ERROR:tasks.cephfs.test_quiesce:inode: { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:17:28.683031+0000", "btime": "2024-04-15T15:17:15.385170+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:17:28.683031+0000", "atime": "2024-04-15T15:17:15.385170+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:17:28.683031+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:17:28.711450+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:17:28.711450+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5222, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5467, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5569, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5856, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5880, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } 2024-04-15T15:17:30.248 ERROR:tasks.cephfs.test_quiesce:op: { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:17:28.904741+0000", "age": 0.644488428, "duration": 0.644618305, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:17:28.904741+0000", "event": "initiated" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "header_read" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "throttled" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "all_read" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "dispatched" }, { "time": "2024-04-15T15:17:28.904919+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:17:28.905356+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:17:28.908421+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:17:28.683031+0000 1=1+0) n(v0 rc2024-04-15T15:17:28.711450+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5222=pAsLsXs/-@2,5467=pAsLsXs/-@7,5569=pAsLsXs/-@2,5856=pAsLsXs/-@3,5880=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x558a07817080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:17:28.683031+0000 1=1+0) n(v0 rc2024-04-15T15:17:28.711450+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5222=pAsLsXs/-@2,5467=pAsLsXs/-@7,5569=pAsLsXs/-@2,5856=pAsLsXs/-@3,5880=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x558a07817080]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } } 2024-04-15T15:17:30.248 ERROR:tasks.cephfs.test_quiesce:lock_type: iversion 2024-04-15T15:17:30.249 ERROR:tasks.cephfs.test_quiesce:cache: [ { "ino": 1099511629282, "rdev": 0, "ctime": "2024-04-15T15:17:28.711450+0000", "btime": "2024-04-15T15:17:28.683031+0000", "mode": 33188, "uid": 1000, "gid": 1301, "nlink": 1, "dir_layout": { "dir_hash": 0, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:17:28.686057+0000", "atime": "2024-04-15T15:17:28.686057+0000", "time_warp_seq": 2, "change_attr": 1, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": true, "client_ranges": [], "dirstat": { "version": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "change_attr": 0 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:17:28.711450+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:17:28.711450+0000" }, "version": 6, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 4, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 0, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5856, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [], "is_subvolume": false }, { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:17:28.683031+0000", "btime": "2024-04-15T15:17:15.385170+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:17:28.683031+0000", "atime": "2024-04-15T15:17:15.385170+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:17:28.683031+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:17:28.711450+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:17:28.711450+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5222, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5467, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5569, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5856, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5880, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } ] 2024-04-15T15:17:30.249 ERROR:tasks.cephfs.test_quiesce:ops: { "1099511627779": { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:17:28.904741+0000", "age": 0.644488428, "duration": 0.644618305, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:17:28.904741+0000", "event": "initiated" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "header_read" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "throttled" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "all_read" }, { "time": "2024-04-15T15:17:28.904741+0000", "event": "dispatched" }, { "time": "2024-04-15T15:17:28.904919+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:17:28.905356+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:17:28.908421+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:17:28.683031+0000 1=1+0) n(v0 rc2024-04-15T15:17:28.711450+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5222=pAsLsXs/-@2,5467=pAsLsXs/-@7,5569=pAsLsXs/-@2,5856=pAsLsXs/-@3,5880=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x558a07817080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/71131a81-30da-45dd-ac67-8e3cd459fcda/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:17:28.683031+0000 1=1+0) n(v0 rc2024-04-15T15:17:28.711450+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5222=pAsLsXs/-@2,5467=pAsLsXs/-@7,5569=pAsLsXs/-@2,5856=pAsLsXs/-@3,5880=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x558a07817080]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } }, "1099511629282": { "description": "internal op quiesce_inode:mds.0:3 fp=#0x100000005e2", "initiated_at": "2024-04-15T15:17:28.905512+0000", "age": 0.643717316, "duration": 0.643934171, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce blocked", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 3 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:17:28.905512+0000", "event": "initiated" }, { "time": "2024-04-15T15:17:28.905512+0000", "event": "header_read" }, { "time": "2024-04-15T15:17:28.905512+0000", "event": "throttled" }, { "time": "2024-04-15T15:17:28.905512+0000", "event": "all_read" }, { "time": "2024-04-15T15:17:28.905512+0000", "event": "dispatched" }, { "time": "2024-04-15T15:17:28.905640+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:17:28.906298+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:17:28.908305+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:17:28.908306+0000", "event": "quiesce blocked" } ], "locks": [] } } } 2024-04-15T15:17:30.249 INFO:tasks.cephfs.test_quiesce:making archive directory /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/quiesce 2024-04-15T15:17:30.251 ERROR:tasks.cephfs.test_quiesce:cache written to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/quiesce/cachez10vwy7y 2024-04-15T15:17:30.253 ERROR:tasks.cephfs.test_quiesce:ops written to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/quiesce/opsxijox0x5 2024-04-15T15:17:30.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:17:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.389+0000 7f32d5d9d640 1 Processor -- start 2024-04-15T15:17:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.389+0000 7f32d5d9d640 1 -- start start 2024-04-15T15:17:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.390+0000 7f32d5d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f32d015e990 0x7f32d006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.390+0000 7f32d5d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32d009a9d0 0x7f32d009ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.390+0000 7f32d5d9d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d006e9d0 con 0x7f32d00b4ab0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.390+0000 7f32d5d9d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f32d015ed70 con 0x7f32d009a9d0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.390+0000 7f32d5d9d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f32d006e000 con 0x7f32d015e990 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.390+0000 7f32cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32d009a9d0 0x7f32d009ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f32d009a9d0 0x7f32d009ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45126/0 (socket says 172.21.15.46:45126) 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cffff640 1 -- 172.21.15.46:0/3778800015 learned_addr learned my addr 172.21.15.46:0/3778800015 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cf7fe640 1 --2- 172.21.15.46:0/3778800015 >> v2:172.21.15.78:3300/0 conn(0x7f32d015e990 0x7f32d006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cffff640 1 -- 172.21.15.46:0/3778800015 >> v2:172.21.15.78:3300/0 conn(0x7f32d015e990 msgr2=0x7f32d006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cffff640 1 --2- 172.21.15.46:0/3778800015 >> v2:172.21.15.78:3300/0 conn(0x7f32d015e990 0x7f32d006e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cffff640 1 -- 172.21.15.46:0/3778800015 >> v1:172.21.15.78:6789/0 conn(0x7f32d00b4ab0 legacy=0x7f32d009b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cffff640 1 -- 172.21.15.46:0/3778800015 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d006d3f0 con 0x7f32d009a9d0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.391+0000 7f32cf7fe640 1 --2- 172.21.15.46:0/3778800015 >> v2:172.21.15.78:3300/0 conn(0x7f32d015e990 0x7f32d006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32cffff640 1 --2- 172.21.15.46:0/3778800015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d009ae00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f32d0073540 tx=0x7f32c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f784c1b67d1752 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32ce7fc640 1 -- 172.21.15.46:0/3778800015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8035020 con 0x7f32d009a9d0 2024-04-15T15:17:30.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32ce7fc640 1 -- 172.21.15.46:0/3778800015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c8031870 con 0x7f32d009a9d0 2024-04-15T15:17:30.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32ce7fc640 1 -- 172.21.15.46:0/3778800015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8031c40 con 0x7f32d009a9d0 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 -- 172.21.15.46:0/3778800015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 msgr2=0x7f32d009ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/3778800015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d009ae00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f32d0073540 tx=0x7f32c802b910 comp rx=0 tx=0).stop 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 -- 172.21.15.46:0/3778800015 shutdown_connections 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/3778800015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d009ae00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/3778800015 >> v2:172.21.15.78:3300/0 conn(0x7f32d015e990 0x7f32d006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 -- 172.21.15.46:0/3778800015 >> 172.21.15.46:0/3778800015 conn(0x7f32d00b3eb0 msgr2=0x7f32d00b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 -- 172.21.15.46:0/3778800015 shutdown_connections 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.392+0000 7f32d5d9d640 1 -- 172.21.15.46:0/3778800015 wait complete. 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.393+0000 7f32d5d9d640 1 Processor -- start 2024-04-15T15:17:30.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 -- start start 2024-04-15T15:17:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d00b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f32d00b4ab0 0x7f32d014a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d015e990 0x7f32d00b5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f32d006d9b0 con 0x7f32d009a9d0 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f32d009b340 con 0x7f32d015e990 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.394+0000 7f32d5d9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f32d006dc90 con 0x7f32d00b4ab0 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d00b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d00b7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45132/0 (socket says 172.21.15.46:45132) 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 -- 172.21.15.46:0/1717619305 learned_addr learned my addr 172.21.15.46:0/1717619305 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32ceffd640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f32d00b4ab0 0x7f32d014a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cffff640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d015e990 0x7f32d00b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 -- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f32d00b4ab0 msgr2=0x7f32d014a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f32d00b4ab0 0x7f32d014a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 -- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d015e990 msgr2=0x7f32d00b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d015e990 0x7f32d00b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 -- 172.21.15.46:0/1717619305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d00b6160 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32cf7fe640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d00b7ac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f32c000cf90 tx=0x7f32c000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c001a070 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c000d840 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c000dc10 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d00b6440 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.395+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d00b6a00 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.396+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32d00689e0 con 0x7f32d009a9d0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.397+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f32c001d070 con 0x7f32d009a9d0 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.397+0000 7f32b7fff640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32bc06c9b0 0x7f32bc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.397+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f32c008c550 con 0x7f32d009a9d0 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.398+0000 7f32ceffd640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32bc06c9b0 0x7f32bc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.398+0000 7f32ceffd640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32bc06c9b0 0x7f32bc06ee70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f32d01e9840 tx=0x7f32c4009060 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.402+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f32c005a100 con 0x7f32d009a9d0 2024-04-15T15:17:30.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.624+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f32d00d0ab0 con 0x7f32d009a9d0 2024-04-15T15:17:30.626 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-04-15T15:17:30.624+0000 7fb7c20f9640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:17:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.626+0000 7f32b7fff640 1 -- 172.21.15.46:0/1717619305 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7f32c005e400 con 0x7f32d009a9d0 2024-04-15T15:17:30.627 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32bc06c9b0 msgr2=0x7f32bc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32bc06c9b0 0x7f32bc06ee70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f32d01e9840 tx=0x7f32c4009060 comp rx=0 tx=0).stop 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 msgr2=0x7f32d00b7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d00b7ac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f32c000cf90 tx=0x7f32c000a7d0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 shutdown_connections 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f32bc06c9b0 0x7f32bc06ee70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f32d015e990 0x7f32d00b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f32d00b4ab0 0x7f32d014a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 --2- 172.21.15.46:0/1717619305 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32d009a9d0 0x7f32d00b7ac0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 >> 172.21.15.46:0/1717619305 conn(0x7f32d00b3eb0 msgr2=0x7f32d00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 shutdown_connections 2024-04-15T15:17:30.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.629+0000 7f32d5d9d640 1 -- 172.21.15.46:0/1717619305 wait complete. 2024-04-15T15:17:30.632 INFO:tasks.ceph.mds.c.smithi046.stderr: -9960> 2024-04-15T15:17:05.846+0000 7fa308937640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:17:30.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-15T15:17:30.667 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-04-15T15:17:30.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.748+0000 7fae277ba640 1 Processor -- start 2024-04-15T15:17:30.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.748+0000 7fae277ba640 1 -- start start 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae277ba640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae200b49d0 0x7fae20160af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae277ba640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fae2009b020 0x7fae200ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae277ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae200d14b0 con 0x7fae2009ac40 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae277ba640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fae200b1b70 con 0x7fae200b49d0 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae277ba640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fae200cd550 con 0x7fae2009b020 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae24d55640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae200b49d0 0x7fae20160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae24d55640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fae200b49d0 0x7fae20160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45154/0 (socket says 172.21.15.46:45154) 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae24d55640 1 -- 172.21.15.46:0/2055323109 learned_addr learned my addr 172.21.15.46:0/2055323109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.749+0000 7fae25d57640 1 --2- 172.21.15.46:0/2055323109 >> v2:172.21.15.78:3300/0 conn(0x7fae2009b020 0x7fae200ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.750+0000 7fae24d55640 1 -- 172.21.15.46:0/2055323109 >> v2:172.21.15.78:3300/0 conn(0x7fae2009b020 msgr2=0x7fae200ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:30.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.750+0000 7fae24d55640 1 --2- 172.21.15.46:0/2055323109 >> v2:172.21.15.78:3300/0 conn(0x7fae2009b020 0x7fae200ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.750+0000 7fae24d55640 1 -- 172.21.15.46:0/2055323109 >> v1:172.21.15.78:6789/0 conn(0x7fae2009ac40 legacy=0x7fae200d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.750+0000 7fae24d55640 1 -- 172.21.15.46:0/2055323109 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae200cd830 con 0x7fae200b49d0 2024-04-15T15:17:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.750+0000 7fae24d55640 1 --2- 172.21.15.46:0/2055323109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae200b49d0 0x7fae20160af0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fae20072790 tx=0x7fae1002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897215331bd59245 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae17fff640 1 -- 172.21.15.46:0/2055323109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1002f850 con 0x7fae200b49d0 2024-04-15T15:17:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae17fff640 1 -- 172.21.15.46:0/2055323109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1002f9f0 con 0x7fae200b49d0 2024-04-15T15:17:30.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae17fff640 1 -- 172.21.15.46:0/2055323109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1002fdc0 con 0x7fae200b49d0 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 -- 172.21.15.46:0/2055323109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae200b49d0 msgr2=0x7fae20160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 --2- 172.21.15.46:0/2055323109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae200b49d0 0x7fae20160af0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fae20072790 tx=0x7fae1002ba30 comp rx=0 tx=0).stop 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 -- 172.21.15.46:0/2055323109 shutdown_connections 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 --2- 172.21.15.46:0/2055323109 >> v2:172.21.15.78:3300/0 conn(0x7fae2009b020 0x7fae200ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 --2- 172.21.15.46:0/2055323109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae200b49d0 0x7fae20160af0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 -- 172.21.15.46:0/2055323109 >> 172.21.15.46:0/2055323109 conn(0x7fae200b3eb0 msgr2=0x7fae200b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 -- 172.21.15.46:0/2055323109 shutdown_connections 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.751+0000 7fae277ba640 1 -- 172.21.15.46:0/2055323109 wait complete. 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 Processor -- start 2024-04-15T15:17:30.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 -- start start 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 0x7fae200cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae2009b020 0x7fae200c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae200b49d0 0x7fae200c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fae200ce570 con 0x7fae2009ac40 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fae20161030 con 0x7fae2009b020 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae277ba640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fae200ce6f0 con 0x7fae200b49d0 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae25556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 0x7fae200cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae25556640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 0x7fae200cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45160/0 (socket says 172.21.15.46:45160) 2024-04-15T15:17:30.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.753+0000 7fae25556640 1 -- 172.21.15.46:0/1410386872 learned_addr learned my addr 172.21.15.46:0/1410386872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae24d55640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae2009b020 0x7fae200c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25d57640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae200b49d0 0x7fae200c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25556640 1 -- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae200b49d0 msgr2=0x7fae200c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25556640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae200b49d0 0x7fae200c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25556640 1 -- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae2009b020 msgr2=0x7fae200c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25556640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae2009b020 0x7fae200c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25556640 1 -- 172.21.15.46:0/1410386872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae20150b40 con 0x7fae2009ac40 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25d57640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae200b49d0 0x7fae200c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae25556640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 0x7fae200cbf80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00c3c0 tx=0x7fae1c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0052d0 con 0x7fae2009ac40 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae1c005490 con 0x7fae2009ac40 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c007260 con 0x7fae2009ac40 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae20150e20 con 0x7fae2009ac40 2024-04-15T15:17:30.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.754+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2006cd00 con 0x7fae2009ac40 2024-04-15T15:17:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.755+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae200cdb10 con 0x7fae2009ac40 2024-04-15T15:17:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.756+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fae1c007400 con 0x7fae2009ac40 2024-04-15T15:17:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.756+0000 7fae15ffb640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae0406c980 0x7fae0406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.756+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fae1c08b1c0 con 0x7fae2009ac40 2024-04-15T15:17:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.756+0000 7fae24d55640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae0406c980 0x7fae0406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.757+0000 7fae24d55640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae0406c980 0x7fae0406ee40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae10036000 tx=0x7fae10008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.760+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae200cdb10 con 0x7fae2009ac40 2024-04-15T15:17:30.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:30.986+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fae2006caf0 con 0x7fae2009ac40 2024-04-15T15:17:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.627+0000 7fae15ffb640 1 -- 172.21.15.46:0/1410386872 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v19) ==== 162+0+0 (secure 0 0 0) 0x7fae1c058db0 con 0x7fae2009ac40 2024-04-15T15:17:31.629 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-15T15:17:31.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae0406c980 msgr2=0x7fae0406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae0406c980 0x7fae0406ee40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fae10036000 tx=0x7fae10008030 comp rx=0 tx=0).stop 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 msgr2=0x7fae200cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 0x7fae200cbf80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fae1c00c3c0 tx=0x7fae1c00c880 comp rx=0 tx=0).stop 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 shutdown_connections 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fae0406c980 0x7fae0406ee40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fae200b49d0 0x7fae200c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae2009b020 0x7fae200c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 --2- 172.21.15.46:0/1410386872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae2009ac40 0x7fae200cbf80 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 >> 172.21.15.46:0/1410386872 conn(0x7fae200b3eb0 msgr2=0x7fae200b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 shutdown_connections 2024-04-15T15:17:31.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.631+0000 7fae277ba640 1 -- 172.21.15.46:0/1410386872 wait complete. 2024-04-15T15:17:31.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:31.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.802+0000 7fa567fe1640 1 Processor -- start 2024-04-15T15:17:31.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.802+0000 7fa567fe1640 1 -- start start 2024-04-15T15:17:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa567fe1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fa560072d80 0x7fa560071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa567fe1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5600af3e0 0x7fa5600af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa567fe1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56015db40 con 0x7fa5600d0690 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa567fe1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fa56015cfc0 con 0x7fa5600af3e0 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa567fe1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fa56015d2a0 con 0x7fa560072d80 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56557c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5600af3e0 0x7fa5600af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56657e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa5600d0690 0x7fa56006e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42990/0 (socket says 172.21.15.46:42990) 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56657e640 1 -- 172.21.15.46:0/2941266290 learned_addr learned my addr 172.21.15.46:0/2941266290 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa565d7d640 1 --2- 172.21.15.46:0/2941266290 >> v2:172.21.15.78:3300/0 conn(0x7fa560072d80 0x7fa560071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56557c640 1 -- 172.21.15.46:0/2941266290 >> v2:172.21.15.78:3300/0 conn(0x7fa560072d80 msgr2=0x7fa560071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56557c640 1 --2- 172.21.15.46:0/2941266290 >> v2:172.21.15.78:3300/0 conn(0x7fa560072d80 0x7fa560071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56557c640 1 -- 172.21.15.46:0/2941266290 >> v1:172.21.15.46:6789/0 conn(0x7fa5600d0690 legacy=0x7fa56006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa56557c640 1 -- 172.21.15.46:0/2941266290 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56015d580 con 0x7fa5600af3e0 2024-04-15T15:17:31.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.803+0000 7fa564d7b640 1 -- 172.21.15.46:0/2941266290 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937558124 0 0) 0x7fa56015db40 con 0x7fa5600d0690 2024-04-15T15:17:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.804+0000 7fa56557c640 1 --2- 172.21.15.46:0/2941266290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600af3e0 0x7fa5600af840 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa5600d11c0 tx=0x7fa55002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e084219e03694a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.804+0000 7fa564d7b640 1 -- 172.21.15.46:0/2941266290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5500315a0 con 0x7fa5600af3e0 2024-04-15T15:17:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.804+0000 7fa564d7b640 1 -- 172.21.15.46:0/2941266290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa550037070 con 0x7fa5600af3e0 2024-04-15T15:17:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.804+0000 7fa564d7b640 1 -- 172.21.15.46:0/2941266290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa550031960 con 0x7fa5600af3e0 2024-04-15T15:17:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 -- 172.21.15.46:0/2941266290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600af3e0 msgr2=0x7fa5600af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2941266290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600af3e0 0x7fa5600af840 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa5600d11c0 tx=0x7fa55002b840 comp rx=0 tx=0).stop 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 -- 172.21.15.46:0/2941266290 shutdown_connections 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2941266290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600af3e0 0x7fa5600af840 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2941266290 >> v2:172.21.15.78:3300/0 conn(0x7fa560072d80 0x7fa560071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 -- 172.21.15.46:0/2941266290 >> 172.21.15.46:0/2941266290 conn(0x7fa56006be00 msgr2=0x7fa56006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 -- 172.21.15.46:0/2941266290 shutdown_connections 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.805+0000 7fa567fe1640 1 -- 172.21.15.46:0/2941266290 wait complete. 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.806+0000 7fa567fe1640 1 Processor -- start 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.806+0000 7fa567fe1640 1 -- start start 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa567fe1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa560072d80 0x7fa5600cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa567fe1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5600af3e0 0x7fa56015aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa567fe1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 0x7fa5600b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa567fe1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa56015cd10 con 0x7fa5600d0690 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa567fe1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fa56015cb90 con 0x7fa5600af3e0 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa567fe1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fa5600d0b00 con 0x7fa560072d80 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 0x7fa5600b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 0x7fa5600b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45178/0 (socket says 172.21.15.46:45178) 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 -- 172.21.15.46:0/2540920056 learned_addr learned my addr 172.21.15.46:0/2540920056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa565d7d640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa560072d80 0x7fa5600cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56557c640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5600af3e0 0x7fa56015aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 -- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa560072d80 msgr2=0x7fa5600cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa560072d80 0x7fa5600cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 -- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5600af3e0 msgr2=0x7fa56015aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5600af3e0 0x7fa56015aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56657e640 1 -- 172.21.15.46:0/2540920056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5600b1710 con 0x7fa5600d0690 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.807+0000 7fa56557c640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5600af3e0 0x7fa56015aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.808+0000 7fa56657e640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 0x7fa5600b0b30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa55800c700 tx=0x7fa55800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.808+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558018070 con 0x7fa5600d0690 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.808+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55800ce10 con 0x7fa5600d0690 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.808+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa560151d70 con 0x7fa5600d0690 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.808+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55800a860 con 0x7fa5600d0690 2024-04-15T15:17:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.808+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5600cd330 con 0x7fa5600d0690 2024-04-15T15:17:31.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.809+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5600cce60 con 0x7fa5600d0690 2024-04-15T15:17:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.809+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa55801b070 con 0x7fa5600d0690 2024-04-15T15:17:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.810+0000 7fa5567fc640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa54406c980 0x7fa54406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.810+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa55808b1c0 con 0x7fa5600d0690 2024-04-15T15:17:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.810+0000 7fa565d7d640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa54406c980 0x7fa54406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.811+0000 7fa565d7d640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa54406c980 0x7fa54406ee40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006fd0 tx=0x7fa55c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:31.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:31.814+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa558058e40 con 0x7fa5600d0690 2024-04-15T15:17:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.037+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5600689e0 con 0x7fa5600d0690 2024-04-15T15:17:32.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.038+0000 7fa5567fc640 1 -- 172.21.15.46:0/2540920056 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5169 (secure 0 0 0) 0x7fa55805d140 con 0x7fa5600d0690 2024-04-15T15:17:32.040 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:32.040 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:31.631752+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:17:32.040 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa54406c980 msgr2=0x7fa54406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa54406c980 0x7fa54406ee40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006fd0 tx=0x7fa55c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 msgr2=0x7fa5600b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 0x7fa5600b0b30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa55800c700 tx=0x7fa55800cbc0 comp rx=0 tx=0).stop 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 shutdown_connections 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fa54406c980 0x7fa54406ee40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5600d0690 0x7fa5600b0b30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5600af3e0 0x7fa56015aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 --2- 172.21.15.46:0/2540920056 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fa560072d80 0x7fa5600cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:32.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 >> 172.21.15.46:0/2540920056 conn(0x7fa56006be00 msgr2=0x7fa56015c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 shutdown_connections 2024-04-15T15:17:32.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:32.040+0000 7fa567fe1640 1 -- 172.21.15.46:0/2540920056 wait complete. 2024-04-15T15:17:32.454 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:17:00.418468+0000', 'modified': '2024-04-15T15:17:31.631752+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5946}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5946': {'gid': 5946, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/245964640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 245964640}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 245964640}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:17:32.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5946:up:replay 2024-04-15T15:17:33.456 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.570+0000 7f7e09e60640 1 Processor -- start 2024-04-15T15:17:33.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.571+0000 7f7e09e60640 1 -- start start 2024-04-15T15:17:33.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.571+0000 7f7e09e60640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7e0415e700 0x7f7e040b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.571+0000 7f7e09e60640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 0x7f7e040cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.571+0000 7f7e09e60640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e040b47a0 con 0x7f7e040d2360 2024-04-15T15:17:33.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.571+0000 7f7e09e60640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f7e040b53b0 con 0x7f7e0415e700 2024-04-15T15:17:33.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.571+0000 7f7e09e60640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7e040b4a80 con 0x7f7e040d3270 2024-04-15T15:17:33.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7dfaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7e040d2360 0x7f7e040d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42998/0 (socket says 172.21.15.46:42998) 2024-04-15T15:17:33.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7dfaffd640 1 -- 172.21.15.46:0/944869577 learned_addr learned my addr 172.21.15.46:0/944869577 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:33.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e03fff640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 0x7f7e040cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e037fe640 1 --2- 172.21.15.46:0/944869577 >> v2:172.21.15.78:3300/0 conn(0x7f7e0415e700 0x7f7e040b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1345406447 0 0) 0x7f7e040b47a0 con 0x7f7e040d2360 2024-04-15T15:17:33.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7df0003520 con 0x7f7e040d2360 2024-04-15T15:17:33.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 227799645 0 0) 0x7f7df0003520 con 0x7f7e040d2360 2024-04-15T15:17:33.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 msgr2=0x7f7e040cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:33.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 0x7f7e040cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 >> v2:172.21.15.78:3300/0 conn(0x7f7e0415e700 msgr2=0x7f7e040b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:33.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 --2- 172.21.15.46:0/944869577 >> v2:172.21.15.78:3300/0 conn(0x7f7e0415e700 0x7f7e040b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e040b4d60 con 0x7f7e040d2360 2024-04-15T15:17:33.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f7dec001670 con 0x7f7e040d2360 2024-04-15T15:17:33.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 >> v1:172.21.15.46:6789/0 conn(0x7f7e040d2360 legacy=0x7f7e040d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:33.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7df00039b0 0x7f7df0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7df0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 0x7f7df00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e040b53b0 con 0x7f7df0004800 2024-04-15T15:17:33.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7e040b4a80 con 0x7f7df00039b0 2024-04-15T15:17:33.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7df000b9c0 con 0x7f7e040d3270 2024-04-15T15:17:33.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7df0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e037fe640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7df00039b0 0x7f7df0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.572+0000 7f7e03fff640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 0x7f7df00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.573+0000 7f7dfaffd640 1 -- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 msgr2=0x7f7df00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:33.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.573+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e040d3270 0x7f7df00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.573+0000 7f7dfaffd640 1 -- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7df00039b0 msgr2=0x7f7df0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:33.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.573+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7df00039b0 0x7f7df0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.573+0000 7f7dfaffd640 1 -- 172.21.15.46:0/944869577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df0003520 con 0x7f7df0004800 2024-04-15T15:17:33.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.573+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7df0004c80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c240 tx=0x7f7df0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf622fdd10e30008 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:33.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec007d30 con 0x7f7df0004800 2024-04-15T15:17:33.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dec030070 con 0x7f7df0004800 2024-04-15T15:17:33.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e02ffd640 1 -- 172.21.15.46:0/944869577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec02cd60 con 0x7f7df0004800 2024-04-15T15:17:33.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e09e60640 1 -- 172.21.15.46:0/944869577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 msgr2=0x7f7df0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:33.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e09e60640 1 --2- 172.21.15.46:0/944869577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7df0004c80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7dec00c240 tx=0x7f7df0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:33.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e03fff640 1 -- 172.21.15.46:0/944869577 reap_dead start 2024-04-15T15:17:33.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e09e60640 1 -- 172.21.15.46:0/944869577 shutdown_connections 2024-04-15T15:17:33.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e09e60640 1 -- 172.21.15.46:0/944869577 >> 172.21.15.46:0/944869577 conn(0x7f7e0408bf60 msgr2=0x7f7e040d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:33.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e09e60640 1 -- 172.21.15.46:0/944869577 shutdown_connections 2024-04-15T15:17:33.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.574+0000 7f7e09e60640 1 -- 172.21.15.46:0/944869577 wait complete. 2024-04-15T15:17:33.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.575+0000 7f7e09e60640 1 Processor -- start 2024-04-15T15:17:33.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 -- start start 2024-04-15T15:17:33.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7e040ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e0414b710 0x7f7e040b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e0414bb90 0x7f7e040b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e040b5d80 con 0x7f7df0004800 2024-04-15T15:17:33.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7e040b4450 con 0x7f7e0414bb90 2024-04-15T15:17:33.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.576+0000 7f7e09e60640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7e040d1d40 con 0x7f7e0414b710 2024-04-15T15:17:33.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7e040ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7e040ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45204/0 (socket says 172.21.15.46:45204) 2024-04-15T15:17:33.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 -- 172.21.15.46:0/1251758019 learned_addr learned my addr 172.21.15.46:0/1251758019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:33.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e03fff640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e0414bb90 0x7f7e040b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e0414b710 0x7f7e040b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 -- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e0414b710 msgr2=0x7f7e040b25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:33.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e0414b710 0x7f7e040b25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 -- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e0414bb90 msgr2=0x7f7e040b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:33.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e0414bb90 0x7f7e040b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 -- 172.21.15.46:0/1251758019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e040b9d00 con 0x7f7df0004800 2024-04-15T15:17:33.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e03fff640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e0414bb90 0x7f7e040b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:33.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e037fe640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7e040ca820 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7df4011980 tx=0x7f7df4011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:33.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df400faf0 con 0x7f7df0004800 2024-04-15T15:17:33.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df400fc90 con 0x7f7df0004800 2024-04-15T15:17:33.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e040b9fe0 con 0x7f7df0004800 2024-04-15T15:17:33.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df4014440 con 0x7f7df0004800 2024-04-15T15:17:33.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.577+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e040b7180 con 0x7f7df0004800 2024-04-15T15:17:33.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.578+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e040689e0 con 0x7f7df0004800 2024-04-15T15:17:33.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.579+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7df4012040 con 0x7f7df0004800 2024-04-15T15:17:33.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.579+0000 7f7e00ff9640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7de406c9b0 0x7f7de406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:33.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.579+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f7df408c510 con 0x7f7df0004800 2024-04-15T15:17:33.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.580+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7de406c9b0 0x7f7de406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:33.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.580+0000 7f7dfaffd640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7de406c9b0 0x7f7de406ee70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7dec034000 tx=0x7f7dec007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:33.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.584+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7df4019030 con 0x7f7df0004800 2024-04-15T15:17:33.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.810+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e040ba500 con 0x7f7df0004800 2024-04-15T15:17:33.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.810+0000 7f7e00ff9640 1 -- 172.21.15.46:0/1251758019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5169 (secure 0 0 0) 0x7f7df405a650 con 0x7f7df0004800 2024-04-15T15:17:33.813 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:33.813 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:32.637673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:17:33.814 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7de406c9b0 msgr2=0x7f7de406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7de406c9b0 0x7f7de406ee70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7dec034000 tx=0x7f7dec007000 comp rx=0 tx=0).stop 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 msgr2=0x7f7e040ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7e040ca820 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7df4011980 tx=0x7f7df4011e40 comp rx=0 tx=0).stop 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 shutdown_connections 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7de406c9b0 0x7f7de406ee70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e0414bb90 0x7f7e040b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e0414b710 0x7f7e040b25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 --2- 172.21.15.46:0/1251758019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7df0004800 0x7f7e040ca820 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 >> 172.21.15.46:0/1251758019 conn(0x7f7e0408bf60 msgr2=0x7f7e04160590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 shutdown_connections 2024-04-15T15:17:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:33.812+0000 7f7e09e60640 1 -- 172.21.15.46:0/1251758019 wait complete. 2024-04-15T15:17:34.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:17:00.418468+0000', 'modified': '2024-04-15T15:17:32.637673+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5946}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5946': {'gid': 5946, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.46:6835/245964640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 245964640}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 245964640}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:17:34.229 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5946:up:rejoin 2024-04-15T15:17:35.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:35.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 Processor -- start 2024-04-15T15:17:35.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 -- start start 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15e700 0x7f7e6c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e6c0d2360 0x7f7e6c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6c0b47a0 con 0x7f7e6c15eae0 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6c0b53b0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.344+0000 7f7e73152640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c0b4a80 con 0x7f7e6c15e700 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e6c0d2360 0x7f7e6c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7e6c0d2360 0x7f7e6c0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45214/0 (socket says 172.21.15.46:45214) 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 -- 172.21.15.46:0/2846388708 learned_addr learned my addr 172.21.15.46:0/2846388708 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e70eee640 1 --2- 172.21.15.46:0/2846388708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15e700 0x7f7e6c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 -- 172.21.15.46:0/2846388708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15e700 msgr2=0x7f7e6c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/2846388708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15e700 0x7f7e6c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 -- 172.21.15.46:0/2846388708 >> v1:172.21.15.78:6789/0 conn(0x7f7e6c15eae0 legacy=0x7f7e6c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:35.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.345+0000 7f7e6bfff640 1 -- 172.21.15.46:0/2846388708 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c0b4d60 con 0x7f7e6c0d2360 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/2846388708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c0d2b60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009870 tx=0x7f7e5802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5152bc6658544325 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e6b7fe640 1 -- 172.21.15.46:0/2846388708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58034070 con 0x7f7e6c0d2360 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e6b7fe640 1 -- 172.21.15.46:0/2846388708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e58030840 con 0x7f7e6c0d2360 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e6b7fe640 1 -- 172.21.15.46:0/2846388708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58030c10 con 0x7f7e6c0d2360 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e73152640 1 -- 172.21.15.46:0/2846388708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 msgr2=0x7f7e6c0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e73152640 1 --2- 172.21.15.46:0/2846388708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c0d2b60 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009870 tx=0x7f7e5802b980 comp rx=0 tx=0).stop 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e73152640 1 -- 172.21.15.46:0/2846388708 shutdown_connections 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e73152640 1 --2- 172.21.15.46:0/2846388708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c0d2b60 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e73152640 1 --2- 172.21.15.46:0/2846388708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15e700 0x7f7e6c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.346+0000 7f7e73152640 1 -- 172.21.15.46:0/2846388708 >> 172.21.15.46:0/2846388708 conn(0x7f7e6c08bf60 msgr2=0x7f7e6c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:35.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.347+0000 7f7e73152640 1 -- 172.21.15.46:0/2846388708 shutdown_connections 2024-04-15T15:17:35.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.347+0000 7f7e73152640 1 -- 172.21.15.46:0/2846388708 wait complete. 2024-04-15T15:17:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.348+0000 7f7e73152640 1 Processor -- start 2024-04-15T15:17:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.348+0000 7f7e73152640 1 -- start start 2024-04-15T15:17:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:35.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c15e700 0x7f7e6c0c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15eae0 0x7f7e6c161340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0b5d80 con 0x7f7e6c0d2360 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0b4450 con 0x7f7e6c15e700 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c0d1d40 con 0x7f7e6c15eae0 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c150a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45222/0 (socket says 172.21.15.46:45222) 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 -- 172.21.15.46:0/3149423323 learned_addr learned my addr 172.21.15.46:0/3149423323 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c15e700 0x7f7e6c0c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e716ef640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15eae0 0x7f7e6c161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 -- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15eae0 msgr2=0x7f7e6c161340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15eae0 0x7f7e6c161340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 -- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c15e700 msgr2=0x7f7e6c0c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c15e700 0x7f7e6c0c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 -- 172.21.15.46:0/3149423323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c0c9c00 con 0x7f7e6c0d2360 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e70eee640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c150a50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000d800 tx=0x7f7e6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000b6c0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e60019070 con 0x7f7e6c0d2360 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.349+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c0c8330 con 0x7f7e6c0d2360 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.350+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6000baa0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.350+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c0c88f0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.351+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6c0689e0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.351+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7e6000bc40 con 0x7f7e6c0d2360 2024-04-15T15:17:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.352+0000 7f7e697fa640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7e5006c980 0x7f7e5006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.352+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f7e6008c2a0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.352+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7e5006c980 0x7f7e5006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:35.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.352+0000 7f7e6bfff640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7e5006c980 0x7f7e5006ee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7e58036000 tx=0x7f7e58008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:35.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.356+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e6005a3e0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.579+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e6c0ba500 con 0x7f7e6c0d2360 2024-04-15T15:17:35.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.580+0000 7f7e697fa640 1 -- 172.21.15.46:0/3149423323 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5176 (secure 0 0 0) 0x7f7e60059ce0 con 0x7f7e6c0d2360 2024-04-15T15:17:35.583 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:35.583 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:34.636332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5946,"qdb_cluster":[5946]},"id":2}]} 2024-04-15T15:17:35.583 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-15T15:17:35.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7e5006c980 msgr2=0x7f7e5006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7e5006c980 0x7f7e5006ee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7e58036000 tx=0x7f7e58008030 comp rx=0 tx=0).stop 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 msgr2=0x7f7e6c150a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c150a50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000d800 tx=0x7f7e6000dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 shutdown_connections 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7e5006c980 0x7f7e5006ee40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7e6c15eae0 0x7f7e6c161340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7e6c15e700 0x7f7e6c0c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 --2- 172.21.15.46:0/3149423323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e6c0d2360 0x7f7e6c150a50 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 >> 172.21.15.46:0/3149423323 conn(0x7f7e6c08bf60 msgr2=0x7f7e6c0d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 shutdown_connections 2024-04-15T15:17:35.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:35.582+0000 7f7e73152640 1 -- 172.21.15.46:0/3149423323 wait complete. 2024-04-15T15:17:35.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:17:00.418468+0000', 'modified': '2024-04-15T15:17:34.636332+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5946}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5946': {'gid': 5946, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.46:6835/245964640', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 245964640}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 245964640}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5946, 'qdb_cluster': [5946]} 2024-04-15T15:17:35.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:17:35.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-15T15:17:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.105+0000 7f0b6ce88640 1 Processor -- start 2024-04-15T15:17:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.105+0000 7f0b6ce88640 1 -- start start 2024-04-15T15:17:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6ce88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b680d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6ce88640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 0x7f0b680d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6ce88640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b680d2480 con 0x7f0b680cfd20 2024-04-15T15:17:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6ce88640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0b680d2db0 con 0x7f0b680d1610 2024-04-15T15:17:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6ce88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b680d2760 con 0x7f0b6815e790 2024-04-15T15:17:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b680d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b66d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 0x7f0b680d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b680d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:55616/0 (socket says 172.21.15.46:55616) 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b5ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0b680cfd20 0x7f0b680d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:46318/0 (socket says 172.21.15.46:46318) 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b66d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 0x7f0b680d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:48500/0 (socket says 172.21.15.46:48500) 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 -- 172.21.15.46:0/3456735769 learned_addr learned my addr 172.21.15.46:0/3456735769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 -- 172.21.15.46:0/3456735769 >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 msgr2=0x7f0b680d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 --2- 172.21.15.46:0/3456735769 >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 0x7f0b680d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 -- 172.21.15.46:0/3456735769 >> v1:172.21.15.78:6789/0 conn(0x7f0b680cfd20 legacy=0x7f0b680d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b6659c640 1 -- 172.21.15.46:0/3456735769 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b680d2a40 con 0x7f0b6815e790 2024-04-15T15:17:36.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.106+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/3456735769 >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 0x7f0b680d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.107+0000 7f0b6659c640 1 --2- 172.21.15.46:0/3456735769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b680d3240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009870 tx=0x7f0b5802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=421f9e3bcb1d7a1f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.107+0000 7f0b65d9b640 1 -- 172.21.15.46:0/3456735769 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58034070 con 0x7f0b6815e790 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.107+0000 7f0b65d9b640 1 -- 172.21.15.46:0/3456735769 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b58030840 con 0x7f0b6815e790 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.107+0000 7f0b65d9b640 1 -- 172.21.15.46:0/3456735769 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58030c10 con 0x7f0b6815e790 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 -- 172.21.15.46:0/3456735769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 msgr2=0x7f0b680d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/3456735769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b680d3240 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009870 tx=0x7f0b5802b980 comp rx=0 tx=0).stop 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 -- 172.21.15.46:0/3456735769 shutdown_connections 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/3456735769 >> v2:172.21.15.78:3300/0 conn(0x7f0b680d1610 0x7f0b680d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/3456735769 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b680d3240 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 -- 172.21.15.46:0/3456735769 >> 172.21.15.46:0/3456735769 conn(0x7f0b6808bf60 msgr2=0x7f0b680d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 -- 172.21.15.46:0/3456735769 shutdown_connections 2024-04-15T15:17:36.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.108+0000 7f0b6ce88640 1 -- 172.21.15.46:0/3456735769 wait complete. 2024-04-15T15:17:36.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.109+0000 7f0b6ce88640 1 Processor -- start 2024-04-15T15:17:36.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.109+0000 7f0b6ce88640 1 -- start start 2024-04-15T15:17:36.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6ce88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 0x7f0b680cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6ce88640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b680d1610 0x7f0b68155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6ce88640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b681f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6ce88640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6815ecf0 con 0x7f0b680cfd20 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6ce88640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b6815eb70 con 0x7f0b680d1610 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6ce88640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b680b74f0 con 0x7f0b6815e790 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 0x7f0b680cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 0x7f0b680cc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45228/0 (socket says 172.21.15.46:45228) 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b6659c640 1 -- 172.21.15.46:0/834631971 learned_addr learned my addr 172.21.15.46:0/834631971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b680d1610 0x7f0b68155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.110+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b681f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6659c640 1 -- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 msgr2=0x7f0b681f86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6659c640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b681f86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6659c640 1 -- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b680d1610 msgr2=0x7f0b68155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6659c640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b680d1610 0x7f0b68155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6659c640 1 -- 172.21.15.46:0/834631971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b680ccea0 con 0x7f0b680cfd20 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6659c640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 0x7f0b680cc7c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0b58000c00 tx=0x7f0b580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b5effd640 1 -- 172.21.15.46:0/834631971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58034040 con 0x7f0b680cfd20 2024-04-15T15:17:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b5effd640 1 -- 172.21.15.46:0/834631971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b58039d90 con 0x7f0b680cfd20 2024-04-15T15:17:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b680cd180 con 0x7f0b680cfd20 2024-04-15T15:17:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b5effd640 1 -- 172.21.15.46:0/834631971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58043610 con 0x7f0b680cfd20 2024-04-15T15:17:36.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.111+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b680cd740 con 0x7f0b680cfd20 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.112+0000 7f0b5effd640 1 -- 172.21.15.46:0/834631971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0b5804a060 con 0x7f0b680cfd20 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.113+0000 7f0b5effd640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b4806c980 0x7f0b4806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.113+0000 7f0b5effd640 1 -- 172.21.15.46:0/834631971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0b580addb0 con 0x7f0b680cfd20 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.113+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b4806c980 0x7f0b4806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.114+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b4806c980 0x7f0b4806ee40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0b680c79c0 tx=0x7f0b50007e80 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.114+0000 7f0b6fa78740 1 Processor -- start 2024-04-15T15:17:36.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.114+0000 7f0b6fa78740 1 -- start start 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bbaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0b8b2f0 0x55c1a0bbb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c1a0af59c0 con 0x55c1a0bb7e10 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c1a0a9c650 con 0x55c1a0bb81f0 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55c1a0af1980 con 0x55c1a0b8b2f0 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45238/0 (socket says 172.21.15.46:45238) 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 -- 172.21.15.46:0/2009226061 learned_addr learned my addr 172.21.15.46:0/2009226061 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bbaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0b8b2f0 0x55c1a0bbb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 -- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0b8b2f0 msgr2=0x55c1a0bbb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0b8b2f0 0x55c1a0bbb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 -- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 msgr2=0x55c1a0bbaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bbaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 -- 172.21.15.46:0/2009226061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1a08082d0 con 0x55c1a0bb7e10 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0b8b2f0 0x55c1a0bbb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b6659c640 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8adb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009280 tx=0x7f0b5807b180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69f135590e30f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.115+0000 7f0b47fff640 1 -- 172.21.15.46:0/2009226061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58091020 con 0x55c1a0bb7e10 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b47fff640 1 -- 172.21.15.46:0/2009226061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5807b760 con 0x55c1a0bb7e10 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b47fff640 1 -- 172.21.15.46:0/2009226061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5807fb10 con 0x55c1a0bb7e10 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 -- 172.21.15.46:0/2009226061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 msgr2=0x55c1a0b8adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8adb0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009280 tx=0x7f0b5807b180 comp rx=0 tx=0).stop 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 -- 172.21.15.46:0/2009226061 shutdown_connections 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0b8b2f0 0x55c1a0bbb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bbaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/2009226061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8adb0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 -- 172.21.15.46:0/2009226061 >> 172.21.15.46:0/2009226061 conn(0x55c1a0b8d020 msgr2=0x55c1a0bd5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 -- 172.21.15.46:0/2009226061 shutdown_connections 2024-04-15T15:17:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.116+0000 7f0b6fa78740 1 -- 172.21.15.46:0/2009226061 wait complete. 2024-04-15T15:17:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.117+0000 7f0b6fa78740 1 -- start start 2024-04-15T15:17:36.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.117+0000 7f0b6fa78740 1 Processor -- start 2024-04-15T15:17:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bd52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bd2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0bd3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c1a0a9c650 con 0x55c1a0bb81f0 2024-04-15T15:17:36.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c1a0aafa10 con 0x55c1a0bb7e10 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55c1a0af59c0 con 0x55c1a0bd07a0 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6fa78740 10 client.0 _ll_register_callbacks cb 0x55c1a0aeb250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bd2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bd2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45254/0 (socket says 172.21.15.46:45254) 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 -- 172.21.15.46:0/1349713367 learned_addr learned my addr 172.21.15.46:0/1349713367 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6659c640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bd52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0bd3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 -- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 msgr2=0x55c1a0bd3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0bd3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 -- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 msgr2=0x55c1a0bd52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bd52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 -- 172.21.15.46:0/1349713367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1a08d26b0 con 0x55c1a0bb81f0 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b6659c640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bd52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:36.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bd2de0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0b6808c620 tx=0x7f0b50000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10226a0f4638e92f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.118+0000 7f0b457fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b457fa640 1 -- 172.21.15.46:0/1349713367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50020580 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b5807b760 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b457fa640 1 -- 172.21.15.46:0/1349713367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0b50020720 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b457fa640 1 -- 172.21.15.46:0/1349713367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0b5002e050 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 10 client.5982 fetch_fsmap learned FSMap version 23 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55c1a0bbcd80 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b457fa640 1 -- 172.21.15.46:0/1349713367 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5807b760 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b457fa640 1 -- 172.21.15.46:0/1349713367 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f0b50031020 con 0x55c1a0bb81f0 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 10 client.5982 fetch_fsmap finished waiting for FSMap version 23 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 10 client.5982 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 20 client.5982 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bbde70 0x55c1a0be96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 4 client.5982 mds_command: new command op to 5946 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:17:36.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 --> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1a0b033e0 con 0x55c1a0bbde70 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.119+0000 7f0b6659c640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bbde70 0x55c1a0be96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b457fa640 10 client.5982 ms_handle_connect on v2:172.21.15.46:6833/245964640 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6659c640 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bbde70 0x55c1a0be96d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b86fcf1687813a server_cookie=8f9e9b3d2d4b757e in_seq=0 out_seq=0 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b457fa640 1 -- 172.21.15.46:0/1349713367 <== mds.0 v2:172.21.15.46:6833/245964640 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c1a0b033e0 con 0x55c1a0bbde70 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b457fa640 10 client.5982 handle_command_reply: tid=0 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 client.5982 shutdown 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 msgr2=0x55c1a0bd2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bd2de0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0b6808c620 tx=0x7f0b50000bc0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 shutdown_connections 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bbde70 0x55c1a0be96d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0bd3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb81f0 0x55c1a0bd2de0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b457fa640 0 client.5982 ms_handle_reset on v2:172.21.15.46:6833/245964640 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/1349713367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bd52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.120+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 >> 172.21.15.46:0/1349713367 conn(0x55c1a0b8b2f0 msgr2=0x55c1a0b8fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.121+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 shutdown_connections 2024-04-15T15:17:36.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.121+0000 7f0b6fa78740 1 -- 172.21.15.46:0/1349713367 wait complete. 2024-04-15T15:17:36.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.121+0000 7f0b6fa78740 20 client.5982 trim_cache size 0 max 16384 2024-04-15T15:17:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.139+0000 7f0b6fa78740 1 Processor -- start 2024-04-15T15:17:36.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.139+0000 7f0b6fa78740 1 -- start start 2024-04-15T15:17:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bb81f0 0x55c1a0bcf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 0x55c1a0bcfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b682005a0 con 0x55c1a0bd07a0 2024-04-15T15:17:36.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b681f5080 con 0x55c1a0bb7e10 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b68200060 con 0x55c1a0bb81f0 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 0x55c1a0bcfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 0x55c1a0bcfcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45268/0 (socket says 172.21.15.46:45268) 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 -- 172.21.15.46:0/593053730 learned_addr learned my addr 172.21.15.46:0/593053730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bb81f0 0x55c1a0bcf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6659c640 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 -- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bb81f0 msgr2=0x55c1a0bcf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bb81f0 0x55c1a0bcf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 -- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 msgr2=0x55c1a0b8eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 -- 172.21.15.46:0/593053730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1a0bd5650 con 0x55c1a0bd07a0 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b6659c640 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 0x55c1a0bcfcc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55c1a0b8d830 tx=0x7f0b60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217c7a71e0a41ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b477fe640 1 -- 172.21.15.46:0/593053730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6001b070 con 0x55c1a0bd07a0 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b477fe640 1 -- 172.21.15.46:0/593053730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b60017850 con 0x55c1a0bd07a0 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.140+0000 7f0b477fe640 1 -- 172.21.15.46:0/593053730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60017c40 con 0x55c1a0bd07a0 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 -- 172.21.15.46:0/593053730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 msgr2=0x55c1a0bcfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 0x55c1a0bcfcc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55c1a0b8d830 tx=0x7f0b60009750 comp rx=0 tx=0).stop 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 -- 172.21.15.46:0/593053730 shutdown_connections 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bd07a0 0x55c1a0bcfcc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bb81f0 0x55c1a0bcf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/593053730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb7e10 0x55c1a0b8eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 -- 172.21.15.46:0/593053730 >> 172.21.15.46:0/593053730 conn(0x55c1a0b8b2f0 msgr2=0x55c1a0be82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 -- 172.21.15.46:0/593053730 shutdown_connections 2024-04-15T15:17:36.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 -- 172.21.15.46:0/593053730 wait complete. 2024-04-15T15:17:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.141+0000 7f0b6fa78740 1 -- start start 2024-04-15T15:17:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 Processor -- start 2024-04-15T15:17:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bcf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0c4f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0c4f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c1a0bbdcf0 con 0x55c1a0bb7e10 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x55c1a0bbd820 con 0x55c1a0bb81f0 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x55c1a0bbe380 con 0x55c1a0bd07a0 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6fa78740 10 client.0 _ll_register_callbacks cb 0x7f0b50004b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.142+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bcf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bcf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45278/0 (socket says 172.21.15.46:45278) 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 -- 172.21.15.46:0/4203891730 learned_addr learned my addr 172.21.15.46:0/4203891730 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b66d9d640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0c4f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0c4f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 -- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 msgr2=0x55c1a0c4f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0c4f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 -- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 msgr2=0x55c1a0c4f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0c4f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 -- 172.21.15.46:0/4203891730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1a0bcfe90 con 0x55c1a0bb7e10 2024-04-15T15:17:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0c4f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6659c640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bcf950 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0b58077b70 tx=0x7f0b5809a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5343ac49b59665 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b457fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b457fa640 1 -- 172.21.15.46:0/4203891730 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58091040 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b457fa640 1 -- 172.21.15.46:0/4203891730 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0b58083eb0 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b60017850 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b457fa640 1 -- 172.21.15.46:0/4203891730 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f0b5807b760 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6fa78740 10 client.5988 fetch_fsmap learned FSMap version 23 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x55c1a0c50f10 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b457fa640 1 -- 172.21.15.46:0/4203891730 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b60017850 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b457fa640 1 -- 172.21.15.46:0/4203891730 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f0b580aa050 con 0x55c1a0bb7e10 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6fa78740 10 client.5988 fetch_fsmap finished waiting for FSMap version 23 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6fa78740 10 client.5988 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.143+0000 7f0b6fa78740 20 client.5988 populate_metadata read hostname 'smithi046' 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bd9780 0x55c1a0bd9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 4 client.5988 mds_command: new command op to 5946 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 --> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] -- command(tid 0: {"prefix": "status"}) -- 0x55c1a0afd250 con 0x55c1a0bd9780 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bd9780 0x55c1a0bd9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b5ffff640 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bd9780 0x55c1a0bd9b60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea0f9e12266aa96d server_cookie=25617acf3006167a in_seq=0 out_seq=0 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b457fa640 10 client.5988 ms_handle_connect on v2:172.21.15.46:6833/245964640 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b457fa640 1 -- 172.21.15.46:0/4203891730 <== mds.0 v2:172.21.15.46:6833/245964640 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55c1a0afd250 con 0x55c1a0bd9780 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b457fa640 10 client.5988 handle_command_reply: tid=0 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 client.5988 shutdown 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 msgr2=0x55c1a0bcf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bcf950 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0b58077b70 tx=0x7f0b5809a000 comp rx=0 tx=0).stop 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 shutdown_connections 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:6833/245964640,v1:172.21.15.46:6835/245964640] conn(0x55c1a0bd9780 0x55c1a0bd9b60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x55c1a0bd07a0 0x55c1a0c4f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c1a0bb81f0 0x55c1a0c4f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b457fa640 0 client.5988 ms_handle_reset on v2:172.21.15.46:6833/245964640 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 --2- 172.21.15.46:0/4203891730 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c1a0bb7e10 0x55c1a0bcf950 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.144+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 >> 172.21.15.46:0/4203891730 conn(0x55c1a0b8b2f0 msgr2=0x55c1a0b863e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 shutdown_connections 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6fa78740 1 -- 172.21.15.46:0/4203891730 wait complete. 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6fa78740 20 client.5988 trim_cache size 0 max 16384 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "95f1e89e-6559-430a-8622-bd018dcc0afd", 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5946, 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-15T15:17:36.154 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.5096585960000004, 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 42, 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 42, 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 5.4783632579999999 2024-04-15T15:17:36.155 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b4806c980 msgr2=0x7f0b4806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b4806c980 0x7f0b4806ee40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0b680c79c0 tx=0x7f0b50007e80 comp rx=0 tx=0).stop 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 msgr2=0x7f0b680cc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 0x7f0b680cc7c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0b58000c00 tx=0x7f0b580046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 shutdown_connections 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b4806c980 0x7f0b4806ee40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b6815e790 0x7f0b681f86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b680d1610 0x7f0b68155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 --2- 172.21.15.46:0/834631971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b680cfd20 0x7f0b680cc7c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 >> 172.21.15.46:0/834631971 conn(0x7f0b6808bf60 msgr2=0x7f0b680b8e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 shutdown_connections 2024-04-15T15:17:36.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.145+0000 7f0b6ce88640 1 -- 172.21.15.46:0/834631971 wait complete. 2024-04-15T15:17:36.159 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:17:36.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:17:36.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:17:36.243 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:17:36.244 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:36.244 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:36.244 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:36.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:36.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.402+0000 7fd56a2fb640 1 Processor -- start 2024-04-15T15:17:36.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.402+0000 7fd56a2fb640 1 -- start start 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd56a2fb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd564072d80 0x7fd564071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd56a2fb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd5640af3e0 0x7fd5640af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd56a2fb640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd5640d0690 0x7fd5640d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd56a2fb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd56415db40 con 0x7fd564072d80 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd56a2fb640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd56415cfc0 con 0x7fd5640d0690 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd56a2fb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd56415d2a0 con 0x7fd5640af3e0 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd564072d80 0x7fd564071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd564072d80 0x7fd564071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45286/0 (socket says 172.21.15.46:45286) 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 -- 172.21.15.46:0/1879026258 learned_addr learned my addr 172.21.15.46:0/1879026258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 -- 172.21.15.46:0/1879026258 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd5640af3e0 msgr2=0x7fd5640af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 --2- 172.21.15.46:0/1879026258 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd5640af3e0 0x7fd5640af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 -- 172.21.15.46:0/1879026258 >> v2:172.21.15.78:3300/0 conn(0x7fd5640d0690 msgr2=0x7fd5640d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd568898640 1 --2- 172.21.15.46:0/1879026258 >> v2:172.21.15.78:3300/0 conn(0x7fd5640d0690 0x7fd5640d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 --2- 172.21.15.46:0/1879026258 >> v2:172.21.15.78:3300/0 conn(0x7fd5640d0690 0x7fd5640d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.403+0000 7fd563fff640 1 -- 172.21.15.46:0/1879026258 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56415d580 con 0x7fd564072d80 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.404+0000 7fd563fff640 1 --2- 172.21.15.46:0/1879026258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564072d80 0x7fd564071d10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd5641600c0 tx=0x7fd55402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f01fc5abab8962 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.404+0000 7fd562ffd640 1 -- 172.21.15.46:0/1879026258 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55402e070 con 0x7fd564072d80 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.404+0000 7fd562ffd640 1 -- 172.21.15.46:0/1879026258 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd554037070 con 0x7fd564072d80 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.404+0000 7fd562ffd640 1 -- 172.21.15.46:0/1879026258 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554031a00 con 0x7fd564072d80 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 -- 172.21.15.46:0/1879026258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564072d80 msgr2=0x7fd564071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/1879026258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564072d80 0x7fd564071d10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd5641600c0 tx=0x7fd55402b880 comp rx=0 tx=0).stop 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 -- 172.21.15.46:0/1879026258 shutdown_connections 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/1879026258 >> v2:172.21.15.78:3300/0 conn(0x7fd5640d0690 0x7fd5640d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/1879026258 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd5640af3e0 0x7fd5640af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/1879026258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd564072d80 0x7fd564071d10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 -- 172.21.15.46:0/1879026258 >> 172.21.15.46:0/1879026258 conn(0x7fd56406be00 msgr2=0x7fd56406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 -- 172.21.15.46:0/1879026258 shutdown_connections 2024-04-15T15:17:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.405+0000 7fd56a2fb640 1 -- 172.21.15.46:0/1879026258 wait complete. 2024-04-15T15:17:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.406+0000 7fd56a2fb640 1 Processor -- start 2024-04-15T15:17:36.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.406+0000 7fd56a2fb640 1 -- start start 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd56a2fb640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd564072d80 0x7fd5641553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd56a2fb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 0x7fd56409dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd56a2fb640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5640d0690 0x7fd5640cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd56a2fb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd56415cd10 con 0x7fd5640af3e0 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd56a2fb640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd56415cb90 con 0x7fd5640d0690 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd56a2fb640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd56406e060 con 0x7fd564072d80 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 0x7fd56409dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 0x7fd56409dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45300/0 (socket says 172.21.15.46:45300) 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 -- 172.21.15.46:0/3913148280 learned_addr learned my addr 172.21.15.46:0/3913148280 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd563fff640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd564072d80 0x7fd5641553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd568898640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5640d0690 0x7fd5640cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 -- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd564072d80 msgr2=0x7fd5641553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd564072d80 0x7fd5641553a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 -- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5640d0690 msgr2=0x7fd5640cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5640d0690 0x7fd5640cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd5637fe640 1 -- 172.21.15.46:0/3913148280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56409b3b0 con 0x7fd5640af3e0 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.407+0000 7fd568898640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5640d0690 0x7fd5640cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.408+0000 7fd5637fe640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 0x7fd56409dc60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd55800b4a0 tx=0x7fd55800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.408+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558014070 con 0x7fd5640af3e0 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.408+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd558010610 con 0x7fd5640af3e0 2024-04-15T15:17:36.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.408+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56409b690 con 0x7fd5640af3e0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.408+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558010a00 con 0x7fd5640af3e0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.408+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd56409bc50 con 0x7fd5640af3e0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.409+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5640ad420 con 0x7fd5640af3e0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.409+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd558017070 con 0x7fd5640af3e0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.410+0000 7fd560ff9640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd55006c980 0x7fd55006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.410+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd55808b850 con 0x7fd5640af3e0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.410+0000 7fd563fff640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd55006c980 0x7fd55006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.410+0000 7fd563fff640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd55006c980 0x7fd55006ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd554036000 tx=0x7fd55403d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.414+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd558059a90 con 0x7fd5640af3e0 2024-04-15T15:17:36.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.599+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5640689e0 con 0x7fd5640af3e0 2024-04-15T15:17:36.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.601+0000 7fd560ff9640 1 -- 172.21.15.46:0/3913148280 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7fd558059390 con 0x7fd5640af3e0 2024-04-15T15:17:36.604 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:36.604 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:36.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.602+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd55006c980 msgr2=0x7fd55006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.602+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd55006c980 0x7fd55006ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd554036000 tx=0x7fd55403d040 comp rx=0 tx=0).stop 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.602+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 msgr2=0x7fd56409dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.602+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 0x7fd56409dc60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd55800b4a0 tx=0x7fd55800b960 comp rx=0 tx=0).stop 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 shutdown_connections 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd55006c980 0x7fd55006ee40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5640d0690 0x7fd5640cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5640af3e0 0x7fd56409dc60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 --2- 172.21.15.46:0/3913148280 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd564072d80 0x7fd5641553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 >> 172.21.15.46:0/3913148280 conn(0x7fd56406be00 msgr2=0x7fd564074d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 shutdown_connections 2024-04-15T15:17:36.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.603+0000 7fd56a2fb640 1 -- 172.21.15.46:0/3913148280 wait complete. 2024-04-15T15:17:36.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:36.617 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:36.638 INFO:tasks.ceph.mds.c.smithi046.stderr:2024-04-15T15:17:36.637+0000 7f41e34a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:17:36.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.722+0000 7fd50f772640 1 Processor -- start 2024-04-15T15:17:36.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.722+0000 7fd50f772640 1 -- start start 2024-04-15T15:17:36.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50f772640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd50815eac0 0x7fd508070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50f772640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd50809b710 0x7fd5080d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50f772640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50809bc10 con 0x7fd50809a9d0 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50f772640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd5080706c0 con 0x7fd50809b710 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50f772640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd5080b1920 con 0x7fd50815eac0 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50cd0d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd50809b710 0x7fd5080d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50dd0f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd50809a9d0 0x7fd5080cca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43006/0 (socket says 172.21.15.46:43006) 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.723+0000 7fd50dd0f640 1 -- 172.21.15.46:0/3027395775 learned_addr learned my addr 172.21.15.46:0/3027395775 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50d50e640 1 --2- 172.21.15.46:0/3027395775 >> v2:172.21.15.78:3300/0 conn(0x7fd50815eac0 0x7fd508070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50cd0d640 1 -- 172.21.15.46:0/3027395775 >> v2:172.21.15.78:3300/0 conn(0x7fd50815eac0 msgr2=0x7fd508070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50cd0d640 1 --2- 172.21.15.46:0/3027395775 >> v2:172.21.15.78:3300/0 conn(0x7fd50815eac0 0x7fd508070180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50cd0d640 1 -- 172.21.15.46:0/3027395775 >> v1:172.21.15.46:6789/0 conn(0x7fd50809a9d0 legacy=0x7fd5080cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50cd0d640 1 -- 172.21.15.46:0/3027395775 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5080d0960 con 0x7fd50809b710 2024-04-15T15:17:36.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50d50e640 1 --2- 172.21.15.46:0/3027395775 >> v2:172.21.15.78:3300/0 conn(0x7fd50815eac0 0x7fd508070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:36.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.724+0000 7fd50cd0d640 1 --2- 172.21.15.46:0/3027395775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809b710 0x7fd5080d1440 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd5081636f0 tx=0x7fd4fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ce176dbb928150 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd507fff640 1 -- 172.21.15.46:0/3027395775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0315a0 con 0x7fd50809b710 2024-04-15T15:17:36.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd507fff640 1 -- 172.21.15.46:0/3027395775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4fc037070 con 0x7fd50809b710 2024-04-15T15:17:36.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd507fff640 1 -- 172.21.15.46:0/3027395775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc031960 con 0x7fd50809b710 2024-04-15T15:17:36.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 -- 172.21.15.46:0/3027395775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809b710 msgr2=0x7fd5080d1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 --2- 172.21.15.46:0/3027395775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809b710 0x7fd5080d1440 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd5081636f0 tx=0x7fd4fc02b840 comp rx=0 tx=0).stop 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 -- 172.21.15.46:0/3027395775 shutdown_connections 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 --2- 172.21.15.46:0/3027395775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809b710 0x7fd5080d1440 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 --2- 172.21.15.46:0/3027395775 >> v2:172.21.15.78:3300/0 conn(0x7fd50815eac0 0x7fd508070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 -- 172.21.15.46:0/3027395775 >> 172.21.15.46:0/3027395775 conn(0x7fd5080b3ec0 msgr2=0x7fd5080b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 -- 172.21.15.46:0/3027395775 shutdown_connections 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.725+0000 7fd50f772640 1 -- 172.21.15.46:0/3027395775 wait complete. 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 Processor -- start 2024-04-15T15:17:36.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 -- start start 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 0x7fd50815b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd50809b710 0x7fd5081495b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd50815eac0 0x7fd5080b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5080d1980 con 0x7fd50809a9d0 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd5080d0fb0 con 0x7fd50815eac0 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50f772640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd50815eea0 con 0x7fd50809b710 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50d50e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 0x7fd50815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50d50e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 0x7fd50815b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45312/0 (socket says 172.21.15.46:45312) 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.727+0000 7fd50d50e640 1 -- 172.21.15.46:0/3584362329 learned_addr learned my addr 172.21.15.46:0/3584362329 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50cd0d640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd50809b710 0x7fd5081495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50dd0f640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd50815eac0 0x7fd5080b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50d50e640 1 -- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd50809b710 msgr2=0x7fd5081495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50d50e640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd50809b710 0x7fd5081495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50d50e640 1 -- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd50815eac0 msgr2=0x7fd5080b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50d50e640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd50815eac0 0x7fd5080b7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50d50e640 1 -- 172.21.15.46:0/3584362329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd508153bc0 con 0x7fd50809a9d0 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50d50e640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 0x7fd50815b660 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd50000e7e0 tx=0x7fd50000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50000f040 con 0x7fd50809a9d0 2024-04-15T15:17:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd500004030 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50815a300 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500004400 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.728+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd50815a8a0 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.729+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd50815b2a0 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.730+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd50000c6d0 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.730+0000 7fd505ffb640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd4f406c9b0 0x7fd4f406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.730+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd50008bc60 con 0x7fd50809a9d0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.730+0000 7fd50cd0d640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd4f406c9b0 0x7fd4f406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.731+0000 7fd50cd0d640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd4f406c9b0 0x7fd4f406ee70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc036000 tx=0x7fd4fc03d040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.734+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd508153bc0 con 0x7fd50809a9d0 2024-04-15T15:17:36.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.920+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5080690f0 con 0x7fd50809a9d0 2024-04-15T15:17:36.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.921+0000 7fd505ffb640 1 -- 172.21.15.46:0/3584362329 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fd500017020 con 0x7fd50809a9d0 2024-04-15T15:17:36.924 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd4f406c9b0 msgr2=0x7fd4f406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd4f406c9b0 0x7fd4f406ee70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc036000 tx=0x7fd4fc03d040 comp rx=0 tx=0).stop 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 msgr2=0x7fd50815b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 0x7fd50815b660 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd50000e7e0 tx=0x7fd50000eca0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 shutdown_connections 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd4f406c9b0 0x7fd4f406ee70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd50815eac0 0x7fd5080b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd50809b710 0x7fd5081495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 --2- 172.21.15.46:0/3584362329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd50809a9d0 0x7fd50815b660 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:36.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 >> 172.21.15.46:0/3584362329 conn(0x7fd5080b3ec0 msgr2=0x7fd50806da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:36.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 shutdown_connections 2024-04-15T15:17:36.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:36.923+0000 7fd50f772640 1 -- 172.21.15.46:0/3584362329 wait complete. 2024-04-15T15:17:36.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:37.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.083+0000 7fb5b0c35640 1 Processor -- start 2024-04-15T15:17:37.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.083+0000 7fb5b0c35640 1 -- start start 2024-04-15T15:17:37.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5b0c35640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5ac15ecb0 0x7fb5ac0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5b0c35640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5ac0b10d0 con 0x7fb5ac0b4350 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5b0c35640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5ac0af920 con 0x7fb5ac0b1d30 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5b0c35640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb5ac15f090 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5aa59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5ac15ecb0 0x7fb5ac0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5a1d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb5ac0b4350 0x7fb5ac0b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43008/0 (socket says 172.21.15.46:43008) 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5aa59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5ac15ecb0 0x7fb5ac0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45324/0 (socket says 172.21.15.46:45324) 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5a1d9b640 1 -- 172.21.15.46:0/2562640787 learned_addr learned my addr 172.21.15.46:0/2562640787 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5aa59c640 1 -- 172.21.15.46:0/2562640787 >> v1:172.21.15.78:6789/0 conn(0x7fb5ac0b1d30 legacy=0x7fb5ac0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5aa59c640 1 -- 172.21.15.46:0/2562640787 >> v1:172.21.15.46:6789/0 conn(0x7fb5ac0b4350 legacy=0x7fb5ac0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/2562640787 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205745393 0 0) 0x7fb5ac0b10d0 con 0x7fb5ac0b4350 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5aa59c640 1 -- 172.21.15.46:0/2562640787 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac0afc00 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/2562640787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b0b90 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0097d0 tx=0x7fb59c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec1f56644a7e886 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/2562640787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c034070 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/2562640787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c030610 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/2562640787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0309e0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5b0c35640 1 -- 172.21.15.46:0/2562640787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 msgr2=0x7fb5ac0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.084+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/2562640787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b0b90 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0097d0 tx=0x7fb59c02b960 comp rx=0 tx=0).stop 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.085+0000 7fb5b0c35640 1 -- 172.21.15.46:0/2562640787 shutdown_connections 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.085+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/2562640787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b0b90 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.085+0000 7fb5b0c35640 1 -- 172.21.15.46:0/2562640787 >> 172.21.15.46:0/2562640787 conn(0x7fb5ac0b6870 msgr2=0x7fb5ac0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.085+0000 7fb5b0c35640 1 -- 172.21.15.46:0/2562640787 shutdown_connections 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.085+0000 7fb5b0c35640 1 -- 172.21.15.46:0/2562640787 wait complete. 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.086+0000 7fb5b0c35640 1 Processor -- start 2024-04-15T15:17:37.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.086+0000 7fb5b0c35640 1 -- start start 2024-04-15T15:17:37.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5b0c35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac0b1d30 0x7fb5ac158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5b0c35640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5ac0b4350 0x7fb5ac156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5b0c35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5b0c35640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac0b47c0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5b0c35640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac0cdf80 con 0x7fb5ac0b1d30 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5b0c35640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac0cb720 con 0x7fb5ac0b4350 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45334/0 (socket says 172.21.15.46:45334) 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 -- 172.21.15.46:0/43073967 learned_addr learned my addr 172.21.15.46:0/43073967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac0b1d30 0x7fb5ac158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5ac0b4350 0x7fb5ac156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 -- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5ac0b4350 msgr2=0x7fb5ac156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5ac0b4350 0x7fb5ac156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 -- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac0b1d30 msgr2=0x7fb5ac158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac0b1d30 0x7fb5ac158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 -- 172.21.15.46:0/43073967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac0cba00 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac0b1d30 0x7fb5ac158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.087+0000 7fb5aad9d640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b4ce0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c780 tx=0x7fb5a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.088+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4016020 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.088+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a400a860 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.088+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac0cbce0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.088+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400acd0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.088+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac0cc2a0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.089+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ac0689e0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.089+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb5a4019020 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.089+0000 7fb5a37fe640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb58c06c9b0 0x7fb58c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.090+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fb5a408c020 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.090+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb58c06c9b0 0x7fb58c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.090+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb58c06c9b0 0x7fb58c06ee70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb59c036000 tx=0x7fb59c03b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.094+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb5a4020020 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.278+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb5ac0cc470 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.279+0000 7fb5a37fe640 1 -- 172.21.15.46:0/43073967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fb5a4059ca0 con 0x7fb5ac15ecb0 2024-04-15T15:17:37.281 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.280+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb58c06c9b0 msgr2=0x7fb58c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.280+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb58c06c9b0 0x7fb58c06ee70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb59c036000 tx=0x7fb59c03b040 comp rx=0 tx=0).stop 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 msgr2=0x7fb5ac0b4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b4ce0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c780 tx=0x7fb5a400cc40 comp rx=0 tx=0).stop 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 shutdown_connections 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb58c06c9b0 0x7fb58c06ee70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac15ecb0 0x7fb5ac0b4ce0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb5ac0b4350 0x7fb5ac156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 --2- 172.21.15.46:0/43073967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb5ac0b1d30 0x7fb5ac158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 >> 172.21.15.46:0/43073967 conn(0x7fb5ac0b6870 msgr2=0x7fb5ac15df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 shutdown_connections 2024-04-15T15:17:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.281+0000 7fb5b0c35640 1 -- 172.21.15.46:0/43073967 wait complete. 2024-04-15T15:17:37.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.400+0000 7fc6e8775640 1 Processor -- start 2024-04-15T15:17:37.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.400+0000 7fc6e8775640 1 -- start start 2024-04-15T15:17:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e8775640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6e009b880 0x7fc6e0070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e8775640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e00d0da0 0x7fc6e00d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e8775640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e00d05e0 con 0x7fc6e0070940 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e8775640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc6e009bce0 con 0x7fc6e009b880 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e8775640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e0071260 con 0x7fc6e00d0da0 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e5d10640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6e009b880 0x7fc6e0070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e5d10640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6e009b880 0x7fc6e0070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45350/0 (socket says 172.21.15.46:45350) 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.401+0000 7fc6e5d10640 1 -- 172.21.15.46:0/1224032294 learned_addr learned my addr 172.21.15.46:0/1224032294 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.402+0000 7fc6e5d10640 1 -- 172.21.15.46:0/1224032294 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e00d0da0 msgr2=0x7fc6e00d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.402+0000 7fc6e5d10640 1 --2- 172.21.15.46:0/1224032294 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e00d0da0 0x7fc6e00d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.402+0000 7fc6e5d10640 1 -- 172.21.15.46:0/1224032294 >> v1:172.21.15.78:6789/0 conn(0x7fc6e0070940 legacy=0x7fc6e0160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:37.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.402+0000 7fc6e5d10640 1 -- 172.21.15.46:0/1224032294 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e015f720 con 0x7fc6e009b880 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.402+0000 7fc6e5d10640 1 --2- 172.21.15.46:0/1224032294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e009b880 0x7fc6e0070d20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009870 tx=0x7fc6d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc769ffe6e573db server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e550f640 1 -- 172.21.15.46:0/1224032294 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d4035020 con 0x7fc6e009b880 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e550f640 1 -- 172.21.15.46:0/1224032294 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d4031640 con 0x7fc6e009b880 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e550f640 1 -- 172.21.15.46:0/1224032294 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d402d040 con 0x7fc6e009b880 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 -- 172.21.15.46:0/1224032294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e009b880 msgr2=0x7fc6e0070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 --2- 172.21.15.46:0/1224032294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e009b880 0x7fc6e0070d20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4009870 tx=0x7fc6d402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 -- 172.21.15.46:0/1224032294 shutdown_connections 2024-04-15T15:17:37.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 --2- 172.21.15.46:0/1224032294 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e00d0da0 0x7fc6e00d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 --2- 172.21.15.46:0/1224032294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e009b880 0x7fc6e0070d20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 -- 172.21.15.46:0/1224032294 >> 172.21.15.46:0/1224032294 conn(0x7fc6e00b3f40 msgr2=0x7fc6e00b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:37.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 -- 172.21.15.46:0/1224032294 shutdown_connections 2024-04-15T15:17:37.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.403+0000 7fc6e8775640 1 -- 172.21.15.46:0/1224032294 wait complete. 2024-04-15T15:17:37.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.404+0000 7fc6e8775640 1 Processor -- start 2024-04-15T15:17:37.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.404+0000 7fc6e8775640 1 -- start start 2024-04-15T15:17:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e8775640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e0070940 0x7fc6e01f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e8775640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e009b880 0x7fc6e015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e8775640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 0x7fc6e01f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e8775640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e00d0880 con 0x7fc6e00d0da0 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e8775640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e015fce0 con 0x7fc6e009b880 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e8775640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e00d1750 con 0x7fc6e0070940 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e6d12640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 0x7fc6e01f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e6d12640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 0x7fc6e01f83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45360/0 (socket says 172.21.15.46:45360) 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e6d12640 1 -- 172.21.15.46:0/2805732766 learned_addr learned my addr 172.21.15.46:0/2805732766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.405+0000 7fc6e6511640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e0070940 0x7fc6e01f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e5d10640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e009b880 0x7fc6e015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6d12640 1 -- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e0070940 msgr2=0x7fc6e01f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6d12640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e0070940 0x7fc6e01f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6d12640 1 -- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e009b880 msgr2=0x7fc6e015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6d12640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e009b880 0x7fc6e015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6d12640 1 -- 172.21.15.46:0/2805732766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e00b8080 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6511640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e0070940 0x7fc6e01f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e5d10640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e009b880 0x7fc6e015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e6d12640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 0x7fc6e01f83b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000b970 tx=0x7fc6d000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0019070 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d000cb00 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e01f8f90 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0014440 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.406+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e01f4ed0 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.407+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e00689e0 con 0x7fc6e00d0da0 2024-04-15T15:17:37.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.409+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc6d001c070 con 0x7fc6e00d0da0 2024-04-15T15:17:37.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.409+0000 7fc6deffd640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6cc06c980 0x7fc6cc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.409+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc6d008bad0 con 0x7fc6e00d0da0 2024-04-15T15:17:37.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.409+0000 7fc6e6511640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6cc06c980 0x7fc6cc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.410+0000 7fc6e6511640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6cc06c980 0x7fc6cc06ee40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00c9370 tx=0x7fc6d8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.413+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6d0059d10 con 0x7fc6e00d0da0 2024-04-15T15:17:37.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.595+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc6e00cea30 con 0x7fc6e00d0da0 2024-04-15T15:17:37.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.596+0000 7fc6deffd640 1 -- 172.21.15.46:0/2805732766 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fc6d0059610 con 0x7fc6e00d0da0 2024-04-15T15:17:37.599 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6cc06c980 msgr2=0x7fc6cc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6cc06c980 0x7fc6cc06ee40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00c9370 tx=0x7fc6d8009040 comp rx=0 tx=0).stop 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 msgr2=0x7fc6e01f83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 0x7fc6e01f83b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc6d000b970 tx=0x7fc6d000be30 comp rx=0 tx=0).stop 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 shutdown_connections 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6cc06c980 0x7fc6cc06ee40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6e00d0da0 0x7fc6e01f83b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6e009b880 0x7fc6e015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 --2- 172.21.15.46:0/2805732766 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6e0070940 0x7fc6e01f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 >> 172.21.15.46:0/2805732766 conn(0x7fc6e00b3f40 msgr2=0x7fc6e015ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 shutdown_connections 2024-04-15T15:17:37.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.598+0000 7fc6e8775640 1 -- 172.21.15.46:0/2805732766 wait complete. 2024-04-15T15:17:37.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.715+0000 7f06a0da8640 1 Processor -- start 2024-04-15T15:17:37.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.715+0000 7f06a0da8640 1 -- start start 2024-04-15T15:17:37.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.715+0000 7f06a0da8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0d0690 0x7f069c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f06a0da8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f069c0ae280 0x7f069c0ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f06a0da8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f069c0d0fb0 con 0x7f069c0b3160 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f06a0da8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f069c15bda0 con 0x7f069c0ae280 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f06a0da8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f069c15cb50 con 0x7f069c0d0690 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f069c0ae280 0x7f069c0ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f0699d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f069c0b3160 0x7f069c15b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43010/0 (socket says 172.21.15.46:43010) 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f0699d9b640 1 -- 172.21.15.46:0/1033689530 learned_addr learned my addr 172.21.15.46:0/1033689530 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069a59c640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0d0690 0x7f069c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069ad9d640 1 -- 172.21.15.46:0/1033689530 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0d0690 msgr2=0x7f069c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069ad9d640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0d0690 0x7f069c0d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069ad9d640 1 -- 172.21.15.46:0/1033689530 >> v1:172.21.15.46:6789/0 conn(0x7f069c0b3160 legacy=0x7f069c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069ad9d640 1 -- 172.21.15.46:0/1033689530 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c15bfd0 con 0x7f069c0ae280 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069959a640 1 -- 172.21.15.46:0/1033689530 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055730513 0 0) 0x7f069c0d0fb0 con 0x7f069c0b3160 2024-04-15T15:17:37.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.716+0000 7f069a59c640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0d0690 0x7f069c0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.717+0000 7f069ad9d640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0ae280 0x7f069c0ae6e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f06940097a0 tx=0x7f069402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8dff9676d8a124 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.717+0000 7f069959a640 1 -- 172.21.15.46:0/1033689530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694035020 con 0x7f069c0ae280 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.717+0000 7f069959a640 1 -- 172.21.15.46:0/1033689530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0694031830 con 0x7f069c0ae280 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.717+0000 7f069959a640 1 -- 172.21.15.46:0/1033689530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694031c00 con 0x7f069c0ae280 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.717+0000 7f06a0da8640 1 -- 172.21.15.46:0/1033689530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0ae280 msgr2=0x7f069c0ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.717+0000 7f06a0da8640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0ae280 0x7f069c0ae6e0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f06940097a0 tx=0x7f069402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.718+0000 7f06a0da8640 1 -- 172.21.15.46:0/1033689530 shutdown_connections 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.718+0000 7f06a0da8640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0ae280 0x7f069c0ae6e0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.718+0000 7f06a0da8640 1 --2- 172.21.15.46:0/1033689530 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0d0690 0x7f069c0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.718+0000 7f06a0da8640 1 -- 172.21.15.46:0/1033689530 >> 172.21.15.46:0/1033689530 conn(0x7f069c06be00 msgr2=0x7f069c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.718+0000 7f06a0da8640 1 -- 172.21.15.46:0/1033689530 shutdown_connections 2024-04-15T15:17:37.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.718+0000 7f06a0da8640 1 -- 172.21.15.46:0/1033689530 wait complete. 2024-04-15T15:17:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.719+0000 7f06a0da8640 1 Processor -- start 2024-04-15T15:17:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.719+0000 7f06a0da8640 1 -- start start 2024-04-15T15:17:37.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0ae280 0x7f069c0ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 0x7f069c15dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f069c0d0690 0x7f069c0cbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f069c15c870 con 0x7f069c0b3160 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f069c15c590 con 0x7f069c0d0690 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f069c0b4fb0 con 0x7f069c0ae280 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 0x7f069c15dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 0x7f069c15dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45382/0 (socket says 172.21.15.46:45382) 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 -- 172.21.15.46:0/2214490799 learned_addr learned my addr 172.21.15.46:0/2214490799 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f069ad9d640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f069c0d0690 0x7f069c0cbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f069a59c640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0ae280 0x7f069c0ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 -- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0ae280 msgr2=0x7f069c0ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0ae280 0x7f069c0ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 -- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f069c0d0690 msgr2=0x7f069c0cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f069c0d0690 0x7f069c0cbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 -- 172.21.15.46:0/2214490799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c15e710 con 0x7f069c0b3160 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f069a59c640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0ae280 0x7f069c0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f0699d9b640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 0x7f069c15dec0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f069000c430 tx=0x7f069000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06900074e0 con 0x7f069c0b3160 2024-04-15T15:17:37.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06900047e0 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069c15e9f0 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0690004bd0 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.721+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069c15efb0 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.722+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f069c0689e0 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.722+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0690011070 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.723+0000 7f068affd640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f067c06c980 0x7f067c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.723+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f069008b8d0 con 0x7f069c0b3160 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.723+0000 7f069a59c640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f067c06c980 0x7f067c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:37.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.723+0000 7f069a59c640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f067c06c980 0x7f067c06ee40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f068c0044f0 tx=0x7f068c003a30 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.727+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0690059ba0 con 0x7f069c0b3160 2024-04-15T15:17:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.926+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f069c15ae40 con 0x7f069c0b3160 2024-04-15T15:17:37.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.927+0000 7f068affd640 1 -- 172.21.15.46:0/2214490799 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+120 (secure 0 0 0) 0x7f06900594a0 con 0x7f069c0b3160 2024-04-15T15:17:37.930 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:37.930 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:37.930 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-04-15T15:17:37.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f067c06c980 msgr2=0x7f067c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f067c06c980 0x7f067c06ee40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f068c0044f0 tx=0x7f068c003a30 comp rx=0 tx=0).stop 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 msgr2=0x7f069c15dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 0x7f069c15dec0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f069000c430 tx=0x7f069000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 shutdown_connections 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f067c06c980 0x7f067c06ee40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f069c0d0690 0x7f069c0cbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f069c0b3160 0x7f069c15dec0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 --2- 172.21.15.46:0/2214490799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f069c0ae280 0x7f069c0ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 >> 172.21.15.46:0/2214490799 conn(0x7f069c06be00 msgr2=0x7f069c0d0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 shutdown_connections 2024-04-15T15:17:37.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:37.928+0000 7f06a0da8640 1 -- 172.21.15.46:0/2214490799 wait complete. 2024-04-15T15:17:37.942 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-04-15T15:17:37.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:37.943 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:37.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[55604]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:37.971 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:17:37.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:37.972 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:17:38.139 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:38.139 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:38.139 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:38.160 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:38.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.283+0000 7f0c2c3f4640 1 Processor -- start 2024-04-15T15:17:38.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.283+0000 7f0c2c3f4640 1 -- start start 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2c3f4640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0c2415c8d0 0x7f0c240cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2c3f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409b650 0x7f0c2409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2c3f4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c2409af10 0x7f0c240d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2c3f4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0c240b5b00 con 0x7f0c2409af10 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2c3f4640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0c240d0350 con 0x7f0c2415c8d0 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2c3f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0c240ce100 con 0x7f0c2409b650 2024-04-15T15:17:38.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c2409af10 0x7f0c240d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c2409af10 0x7f0c240d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45392/0 (socket says 172.21.15.46:45392) 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 -- 172.21.15.46:0/4066624850 learned_addr learned my addr 172.21.15.46:0/4066624850 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2998f640 1 --2- 172.21.15.46:0/4066624850 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409b650 0x7f0c2409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a190640 1 --2- 172.21.15.46:0/4066624850 >> v2:172.21.15.78:3300/0 conn(0x7f0c2415c8d0 0x7f0c240cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 -- 172.21.15.46:0/4066624850 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409b650 msgr2=0x7f0c2409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 --2- 172.21.15.46:0/4066624850 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409b650 0x7f0c2409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 -- 172.21.15.46:0/4066624850 >> v2:172.21.15.78:3300/0 conn(0x7f0c2415c8d0 msgr2=0x7f0c240cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 --2- 172.21.15.46:0/4066624850 >> v2:172.21.15.78:3300/0 conn(0x7f0c2415c8d0 0x7f0c240cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a991640 1 -- 172.21.15.46:0/4066624850 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c240b54b0 con 0x7f0c2409af10 2024-04-15T15:17:38.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.284+0000 7f0c2a190640 1 --2- 172.21.15.46:0/4066624850 >> v2:172.21.15.78:3300/0 conn(0x7f0c2415c8d0 0x7f0c240cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:38.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.285+0000 7f0c2a991640 1 --2- 172.21.15.46:0/4066624850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c2409af10 0x7f0c240d0540 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0c2409b620 tx=0x7f0c1c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45a606561769407 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.285+0000 7f0c2918e640 1 -- 172.21.15.46:0/4066624850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c02c9e0 con 0x7f0c2409af10 2024-04-15T15:17:38.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.285+0000 7f0c2918e640 1 -- 172.21.15.46:0/4066624850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c1c02cb80 con 0x7f0c2409af10 2024-04-15T15:17:38.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.285+0000 7f0c2918e640 1 -- 172.21.15.46:0/4066624850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0343f0 con 0x7f0c2409af10 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.285+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/4066624850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c2409af10 msgr2=0x7f0c240d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.285+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/4066624850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c2409af10 0x7f0c240d0540 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0c2409b620 tx=0x7f0c1c02be40 comp rx=0 tx=0).stop 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/4066624850 shutdown_connections 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/4066624850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c2409af10 0x7f0c240d0540 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/4066624850 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409b650 0x7f0c2409a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/4066624850 >> v2:172.21.15.78:3300/0 conn(0x7f0c2415c8d0 0x7f0c240cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/4066624850 >> 172.21.15.46:0/4066624850 conn(0x7f0c240b3fb0 msgr2=0x7f0c240b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:38.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/4066624850 shutdown_connections 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.286+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/4066624850 wait complete. 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.287+0000 7f0c2c3f4640 1 Processor -- start 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.287+0000 7f0c2c3f4640 1 -- start start 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.287+0000 7f0c2c3f4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409af10 0x7f0c24154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2409b650 0x7f0c240ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 0x7f0c24155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0c240d0c00 con 0x7f0c240cab20 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0c240d0a80 con 0x7f0c2409b650 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0c240d0d80 con 0x7f0c2409af10 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 0x7f0c24155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 0x7f0c24155a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45406/0 (socket says 172.21.15.46:45406) 2024-04-15T15:17:38.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 -- 172.21.15.46:0/3643203509 learned_addr learned my addr 172.21.15.46:0/3643203509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a190640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409af10 0x7f0c24154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2998f640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2409b650 0x7f0c240ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 -- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409af10 msgr2=0x7f0c24154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409af10 0x7f0c24154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 -- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2409b650 msgr2=0x7f0c240ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2409b650 0x7f0c240ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 -- 172.21.15.46:0/3643203509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c24156670 con 0x7f0c240cab20 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2998f640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2409b650 0x7f0c240ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2a991640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 0x7f0c24155a20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009f40 tx=0x7f0c1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c043020 con 0x7f0c240cab20 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c1c03e450 con 0x7f0c240cab20 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c24156e90 con 0x7f0c240cab20 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c03e820 con 0x7f0c240cab20 2024-04-15T15:17:38.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.288+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c24157450 con 0x7f0c240cab20 2024-04-15T15:17:38.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.289+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c24157650 con 0x7f0c240cab20 2024-04-15T15:17:38.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.290+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0c1c046020 con 0x7f0c240cab20 2024-04-15T15:17:38.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.290+0000 7f0c1affd640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0c0806c980 0x7f0c0806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.291+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0c1c0ae680 con 0x7f0c240cab20 2024-04-15T15:17:38.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.291+0000 7f0c2a190640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0c0806c980 0x7f0c0806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.291+0000 7f0c2a190640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0c0806c980 0x7f0c0806ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000a8d0 tx=0x7f0c20008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.295+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c24157650 con 0x7f0c240cab20 2024-04-15T15:17:38.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.481+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c240690f0 con 0x7f0c240cab20 2024-04-15T15:17:38.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.482+0000 7f0c1affd640 1 -- 172.21.15.46:0/3643203509 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7f0c1c04d030 con 0x7f0c240cab20 2024-04-15T15:17:38.484 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:38.484 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:38.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0c0806c980 msgr2=0x7f0c0806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0c0806c980 0x7f0c0806ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000a8d0 tx=0x7f0c20008040 comp rx=0 tx=0).stop 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 msgr2=0x7f0c24155a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 0x7f0c24155a20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009f40 tx=0x7f0c1c004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 shutdown_connections 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0c0806c980 0x7f0c0806ee40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c240cab20 0x7f0c24155a20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c2409b650 0x7f0c240ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 --2- 172.21.15.46:0/3643203509 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0c2409af10 0x7f0c24154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 >> 172.21.15.46:0/3643203509 conn(0x7f0c240b3fb0 msgr2=0x7f0c240b6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 shutdown_connections 2024-04-15T15:17:38.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.484+0000 7f0c2c3f4640 1 -- 172.21.15.46:0/3643203509 wait complete. 2024-04-15T15:17:38.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:38.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:38.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.604+0000 7f64f1d3d640 1 Processor -- start 2024-04-15T15:17:38.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.604+0000 7f64f1d3d640 1 -- start start 2024-04-15T15:17:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64f1d3d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 0x7f64ec0cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64f1d3d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64ec09ab40 0x7f64ec09afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64f1d3d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64ec15dbd0 con 0x7f64ec1606b0 2024-04-15T15:17:38.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64f1d3d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f64ec0cd6a0 con 0x7f64ec09ab40 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64f1d3d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f64ec0cd980 con 0x7f64ec1602d0 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64ebfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64ec09ab40 0x7f64ec09afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 0x7f64ec0cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64ebfff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64ec09ab40 0x7f64ec09afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:48656/0 (socket says 172.21.15.46:48656) 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64eaffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f64ec1606b0 0x7f64ec0d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:46350/0 (socket says 172.21.15.46:46350) 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64eb7fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 0x7f64ec0cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:55748/0 (socket says 172.21.15.46:55748) 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.605+0000 7f64ebfff640 1 -- 172.21.15.46:0/4268144437 learned_addr learned my addr 172.21.15.46:0/4268144437 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.606+0000 7f64eb7fe640 1 -- 172.21.15.46:0/4268144437 >> v2:172.21.15.78:3300/0 conn(0x7f64ec09ab40 msgr2=0x7f64ec09afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.606+0000 7f64eb7fe640 1 --2- 172.21.15.46:0/4268144437 >> v2:172.21.15.78:3300/0 conn(0x7f64ec09ab40 0x7f64ec09afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.606+0000 7f64eb7fe640 1 -- 172.21.15.46:0/4268144437 >> v1:172.21.15.78:6789/0 conn(0x7f64ec1606b0 legacy=0x7f64ec0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.606+0000 7f64eb7fe640 1 -- 172.21.15.46:0/4268144437 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec0cdc60 con 0x7f64ec1602d0 2024-04-15T15:17:38.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64eb7fe640 1 --2- 172.21.15.46:0/4268144437 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 0x7f64ec0cd000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0097a0 tx=0x7f64dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e172447924156f5d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64ea7fc640 1 -- 172.21.15.46:0/4268144437 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc035020 con 0x7f64ec1602d0 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64ea7fc640 1 -- 172.21.15.46:0/4268144437 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64dc031830 con 0x7f64ec1602d0 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64ea7fc640 1 -- 172.21.15.46:0/4268144437 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc031c00 con 0x7f64ec1602d0 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 -- 172.21.15.46:0/4268144437 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 msgr2=0x7f64ec0cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/4268144437 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 0x7f64ec0cd000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f64dc0097a0 tx=0x7f64dc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 -- 172.21.15.46:0/4268144437 shutdown_connections 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/4268144437 >> v2:172.21.15.78:3300/0 conn(0x7f64ec09ab40 0x7f64ec09afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/4268144437 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1602d0 0x7f64ec0cd000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 -- 172.21.15.46:0/4268144437 >> 172.21.15.46:0/4268144437 conn(0x7f64ec0d2460 msgr2=0x7f64ec0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 -- 172.21.15.46:0/4268144437 shutdown_connections 2024-04-15T15:17:38.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.607+0000 7f64f1d3d640 1 -- 172.21.15.46:0/4268144437 wait complete. 2024-04-15T15:17:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.608+0000 7f64f1d3d640 1 Processor -- start 2024-04-15T15:17:38.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 -- start start 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ec09ab40 0x7f64ec0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 0x7f64ec0c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1606b0 0x7f64ec06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec160b10 con 0x7f64ec1602d0 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f64ec0ce220 con 0x7f64ec09ab40 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64f1d3d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f64ec09b510 con 0x7f64ec1606b0 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.609+0000 7f64eaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 0x7f64ec0c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 0x7f64ec0c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45422/0 (socket says 172.21.15.46:45422) 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 -- 172.21.15.46:0/1013402716 learned_addr learned my addr 172.21.15.46:0/1013402716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64ebfff640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1606b0 0x7f64ec06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eb7fe640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ec09ab40 0x7f64ec0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 -- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1606b0 msgr2=0x7f64ec06d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1606b0 0x7f64ec06d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 -- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ec09ab40 msgr2=0x7f64ec0b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ec09ab40 0x7f64ec0b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 -- 172.21.15.46:0/1013402716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64ec1598f0 con 0x7f64ec1602d0 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64eaffd640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 0x7f64ec0c5050 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c3c0 tx=0x7f64e000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e00074e0 con 0x7f64ec1602d0 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e0007680 con 0x7f64ec1602d0 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64ec159bd0 con 0x7f64ec1602d0 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0004a00 con 0x7f64ec1602d0 2024-04-15T15:17:38.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.610+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64ec0c5590 con 0x7f64ec1602d0 2024-04-15T15:17:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.611+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64ec0689e0 con 0x7f64ec1602d0 2024-04-15T15:17:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.612+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f64e001a020 con 0x7f64ec1602d0 2024-04-15T15:17:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.612+0000 7f64d3fff640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64d806c980 0x7f64d806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.613+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f64e008b380 con 0x7f64ec1602d0 2024-04-15T15:17:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.613+0000 7f64eb7fe640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64d806c980 0x7f64d806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.613+0000 7f64eb7fe640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64d806c980 0x7f64d806ee40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f64dc036000 tx=0x7f64dc008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.617+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64e0059000 con 0x7f64ec1602d0 2024-04-15T15:17:38.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.804+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f64ec075340 con 0x7f64ec1602d0 2024-04-15T15:17:38.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.804+0000 7f64d3fff640 1 -- 172.21.15.46:0/1013402716 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f64e005d300 con 0x7f64ec1602d0 2024-04-15T15:17:38.806 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64d806c980 msgr2=0x7f64d806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64d806c980 0x7f64d806ee40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f64dc036000 tx=0x7f64dc008030 comp rx=0 tx=0).stop 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 msgr2=0x7f64ec0c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 0x7f64ec0c5050 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c3c0 tx=0x7f64e000c880 comp rx=0 tx=0).stop 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 shutdown_connections 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64d806c980 0x7f64d806ee40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64ec1606b0 0x7f64ec06d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64ec1602d0 0x7f64ec0c5050 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 --2- 172.21.15.46:0/1013402716 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64ec09ab40 0x7f64ec0b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 >> 172.21.15.46:0/1013402716 conn(0x7f64ec0d2460 msgr2=0x7f64ec071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 shutdown_connections 2024-04-15T15:17:38.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.806+0000 7f64f1d3d640 1 -- 172.21.15.46:0/1013402716 wait complete. 2024-04-15T15:17:38.819 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:38.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.925+0000 7f05a9663640 1 Processor -- start 2024-04-15T15:17:38.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.925+0000 7f05a9663640 1 -- start start 2024-04-15T15:17:38.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.925+0000 7f05a9663640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05a415e700 0x7f05a40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a9663640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f05a40d2360 0x7f05a40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a9663640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a40b47a0 con 0x7f05a415eae0 2024-04-15T15:17:38.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a9663640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f05a40b53b0 con 0x7f05a415e700 2024-04-15T15:17:38.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a9663640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f05a40b4a80 con 0x7f05a40d2360 2024-04-15T15:17:38.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05a415e700 0x7f05a40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f05a415e700 0x7f05a40b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45430/0 (socket says 172.21.15.46:45430) 2024-04-15T15:17:38.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 -- 172.21.15.46:0/3680502873 learned_addr learned my addr 172.21.15.46:0/3680502873 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:38.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f059a7fc640 1 --2- 172.21.15.46:0/3680502873 >> v2:172.21.15.78:3300/0 conn(0x7f05a40d2360 0x7f05a40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 -- 172.21.15.46:0/3680502873 >> v2:172.21.15.78:3300/0 conn(0x7f05a40d2360 msgr2=0x7f05a40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 --2- 172.21.15.46:0/3680502873 >> v2:172.21.15.78:3300/0 conn(0x7f05a40d2360 0x7f05a40d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 -- 172.21.15.46:0/3680502873 >> v1:172.21.15.78:6789/0 conn(0x7f05a415eae0 legacy=0x7f05a40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.926+0000 7f05a2ffd640 1 -- 172.21.15.46:0/3680502873 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a40b4d60 con 0x7f05a415e700 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.927+0000 7f05a2ffd640 1 --2- 172.21.15.46:0/3680502873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415e700 0x7f05a40b5840 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f05a408c7b0 tx=0x7f059402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26176fe501093191 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.927+0000 7f05a27fc640 1 -- 172.21.15.46:0/3680502873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594004230 con 0x7f05a415e700 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.927+0000 7f05a27fc640 1 -- 172.21.15.46:0/3680502873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05940043d0 con 0x7f05a415e700 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.927+0000 7f05a27fc640 1 -- 172.21.15.46:0/3680502873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594030840 con 0x7f05a415e700 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.927+0000 7f05a9663640 1 -- 172.21.15.46:0/3680502873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415e700 msgr2=0x7f05a40b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.927+0000 7f05a9663640 1 --2- 172.21.15.46:0/3680502873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415e700 0x7f05a40b5840 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f05a408c7b0 tx=0x7f059402b980 comp rx=0 tx=0).stop 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.928+0000 7f05a9663640 1 -- 172.21.15.46:0/3680502873 shutdown_connections 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.928+0000 7f05a9663640 1 --2- 172.21.15.46:0/3680502873 >> v2:172.21.15.78:3300/0 conn(0x7f05a40d2360 0x7f05a40d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.928+0000 7f05a9663640 1 --2- 172.21.15.46:0/3680502873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415e700 0x7f05a40b5840 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.928+0000 7f05a9663640 1 -- 172.21.15.46:0/3680502873 >> 172.21.15.46:0/3680502873 conn(0x7f05a408bf60 msgr2=0x7f05a40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:38.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.928+0000 7f05a9663640 1 -- 172.21.15.46:0/3680502873 shutdown_connections 2024-04-15T15:17:38.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.928+0000 7f05a9663640 1 -- 172.21.15.46:0/3680502873 wait complete. 2024-04-15T15:17:38.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.929+0000 7f05a9663640 1 Processor -- start 2024-04-15T15:17:38.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.929+0000 7f05a9663640 1 -- start start 2024-04-15T15:17:38.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f05a40d2360 0x7f05a414b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05a415e700 0x7f05a40cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 0x7f05a41f7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05a40b5d80 con 0x7f05a415eae0 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f05a40b4450 con 0x7f05a415e700 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f05a40d1d40 con 0x7f05a40d2360 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 0x7f05a41f7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 0x7f05a41f7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45444/0 (socket says 172.21.15.46:45444) 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 -- 172.21.15.46:0/2465457011 learned_addr learned my addr 172.21.15.46:0/2465457011 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f059a7fc640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05a415e700 0x7f05a40cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a2ffd640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f05a40d2360 0x7f05a414b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 -- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f05a40d2360 msgr2=0x7f05a414b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f05a40d2360 0x7f05a414b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 -- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05a415e700 msgr2=0x7f05a40cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05a415e700 0x7f05a40cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 -- 172.21.15.46:0/2465457011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a40c87a0 con 0x7f05a415eae0 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a37fe640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 0x7f05a41f7d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f059c00c6b0 tx=0x7f059c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c003b70 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059c003d30 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a40ce520 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.930+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c005310 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.931+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a40c84c0 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.931+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a40689e0 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.932+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f059c0054b0 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.932+0000 7f059bfff640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f058406c980 0x7f058406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.932+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f059c08bbd0 con 0x7f05a415eae0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.933+0000 7f05a2ffd640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f058406c980 0x7f058406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:38.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.933+0000 7f05a2ffd640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f058406c980 0x7f058406ee40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0594035000 tx=0x7f0594008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:38.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:38.937+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f059c0596c0 con 0x7f05a415eae0 2024-04-15T15:17:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.120+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f05a40ba500 con 0x7f05a415eae0 2024-04-15T15:17:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.120+0000 7f059bfff640 1 -- 172.21.15.46:0/2465457011 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f059c05d9c0 con 0x7f05a415eae0 2024-04-15T15:17:39.123 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f058406c980 msgr2=0x7f058406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f058406c980 0x7f058406ee40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0594035000 tx=0x7f0594008030 comp rx=0 tx=0).stop 2024-04-15T15:17:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 msgr2=0x7f05a41f7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 0x7f05a41f7d80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f059c00c6b0 tx=0x7f059c00cb70 comp rx=0 tx=0).stop 2024-04-15T15:17:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 shutdown_connections 2024-04-15T15:17:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f058406c980 0x7f058406ee40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05a415eae0 0x7f05a41f7d80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f05a415e700 0x7f05a40cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 --2- 172.21.15.46:0/2465457011 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f05a40d2360 0x7f05a414b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 >> 172.21.15.46:0/2465457011 conn(0x7f05a408bf60 msgr2=0x7f05a415f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 shutdown_connections 2024-04-15T15:17:39.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.122+0000 7f05a9663640 1 -- 172.21.15.46:0/2465457011 wait complete. 2024-04-15T15:17:39.135 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.241+0000 7ff1ae450640 1 Processor -- start 2024-04-15T15:17:39.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.241+0000 7ff1ae450640 1 -- start start 2024-04-15T15:17:39.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1ae450640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1a80d3270 0x7ff1a80cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1ae450640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1a80b47a0 con 0x7ff1a80d2360 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1ae450640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1a80b4a80 con 0x7ff1a815e700 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1ae450640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff1a80b53b0 con 0x7ff1a80d3270 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1ac9ed640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff1a80d3270 0x7ff1a80cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1a77fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff1a80d2360 0x7ff1a80d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:43024/0 (socket says 172.21.15.46:43024) 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.242+0000 7ff1a77fe640 1 -- 172.21.15.46:0/986991742 learned_addr learned my addr 172.21.15.46:0/986991742 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.243+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/986991742 >> v1:172.21.15.78:6789/0 conn(0x7ff1a815e700 legacy=0x7ff1a80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.243+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/986991742 >> v1:172.21.15.46:6789/0 conn(0x7ff1a80d2360 legacy=0x7ff1a80d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.243+0000 7ff1a6ffd640 1 -- 172.21.15.46:0/986991742 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495359855 0 0) 0x7ff1a80b47a0 con 0x7ff1a80d2360 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.243+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/986991742 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a80b4d60 con 0x7ff1a80d3270 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ac9ed640 1 --2- 172.21.15.46:0/986991742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a80d3270 0x7ff1a80cf370 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80b5530 tx=0x7ff1a002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4918e83e600c6f25 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1a6ffd640 1 -- 172.21.15.46:0/986991742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0039070 con 0x7ff1a80d3270 2024-04-15T15:17:39.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1a6ffd640 1 -- 172.21.15.46:0/986991742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a002cce0 con 0x7ff1a80d3270 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1a6ffd640 1 -- 172.21.15.46:0/986991742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0034660 con 0x7ff1a80d3270 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 -- 172.21.15.46:0/986991742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a80d3270 msgr2=0x7ff1a80cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 --2- 172.21.15.46:0/986991742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a80d3270 0x7ff1a80cf370 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff1a80b5530 tx=0x7ff1a002bf30 comp rx=0 tx=0).stop 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 -- 172.21.15.46:0/986991742 shutdown_connections 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 --2- 172.21.15.46:0/986991742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a80d3270 0x7ff1a80cf370 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 -- 172.21.15.46:0/986991742 >> 172.21.15.46:0/986991742 conn(0x7ff1a808bf60 msgr2=0x7ff1a80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 -- 172.21.15.46:0/986991742 shutdown_connections 2024-04-15T15:17:39.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.244+0000 7ff1ae450640 1 -- 172.21.15.46:0/986991742 wait complete. 2024-04-15T15:17:39.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 Processor -- start 2024-04-15T15:17:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 -- start start 2024-04-15T15:17:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1a80d2360 0x7ff1a81e7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff1a80d3270 0x7ff1a80cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 0x7ff1a81f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a80d1d40 con 0x7ff1a815e700 2024-04-15T15:17:39.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a80b4450 con 0x7ff1a80d2360 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.246+0000 7ff1ae450640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a80d1ec0 con 0x7ff1a80d3270 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 0x7ff1a81f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 0x7ff1a81f4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45462/0 (socket says 172.21.15.46:45462) 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/3882629799 learned_addr learned my addr 172.21.15.46:0/3882629799 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1a77fe640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff1a80d3270 0x7ff1a80cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1a7fff640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1a80d2360 0x7ff1a81e7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff1a80d3270 msgr2=0x7ff1a80cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff1a80d3270 0x7ff1a80cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1a80d2360 msgr2=0x7ff1a81e7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1a80d2360 0x7ff1a81e7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 -- 172.21.15.46:0/3882629799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a80ccc80 con 0x7ff1a815e700 2024-04-15T15:17:39.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ac9ed640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 0x7ff1a81f4170 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0037040 tx=0x7ff1a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a0039050 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a0045070 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a80ccf60 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a003e810 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.247+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a80cd800 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.248+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a80689e0 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.249+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff1a003e9b0 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.249+0000 7ff1a4ff9640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff19406c980 0x7ff19406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.249+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7ff1a00aebe0 con 0x7ff1a815e700 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.249+0000 7ff1a7fff640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff19406c980 0x7ff19406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.250+0000 7ff1a7fff640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff19406c980 0x7ff19406ee40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff198000fb0 tx=0x7ff198009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:39.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.253+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1a007cf40 con 0x7ff1a815e700 2024-04-15T15:17:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.437+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff1a815dfb0 con 0x7ff1a815e700 2024-04-15T15:17:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.438+0000 7ff1a4ff9640 1 -- 172.21.15.46:0/3882629799 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7ff1a007c840 con 0x7ff1a815e700 2024-04-15T15:17:39.440 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff19406c980 msgr2=0x7ff19406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff19406c980 0x7ff19406ee40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff198000fb0 tx=0x7ff198009040 comp rx=0 tx=0).stop 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 msgr2=0x7ff1a81f4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 0x7ff1a81f4170 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0037040 tx=0x7ff1a0004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 shutdown_connections 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff19406c980 0x7ff19406ee40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a815e700 0x7ff1a81f4170 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff1a80d3270 0x7ff1a80cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 --2- 172.21.15.46:0/3882629799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff1a80d2360 0x7ff1a81e7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.439+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 >> 172.21.15.46:0/3882629799 conn(0x7ff1a808bf60 msgr2=0x7ff1a80d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.440+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 shutdown_connections 2024-04-15T15:17:39.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.440+0000 7ff1ae450640 1 -- 172.21.15.46:0/3882629799 wait complete. 2024-04-15T15:17:39.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.557+0000 7f10cdb32640 1 Processor -- start 2024-04-15T15:17:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.557+0000 7f10cdb32640 1 -- start start 2024-04-15T15:17:39.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10cdb32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8070940 0x7f10c809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10cdb32640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f10c80d0da0 0x7f10c80d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10cdb32640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10c80d05e0 con 0x7f10c8160310 2024-04-15T15:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10cdb32640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f10c815f720 con 0x7f10c80d0da0 2024-04-15T15:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10cdb32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f10c815fa00 con 0x7f10c8070940 2024-04-15T15:17:39.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10c7fff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f10c80d0da0 0x7f10c80d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10c77fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8070940 0x7f10c809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10c6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f10c8160310 0x7f10c8070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:40010/0 (socket says 172.21.15.46:40010) 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.558+0000 7f10c6ffd640 1 -- 172.21.15.46:0/2828155848 learned_addr learned my addr 172.21.15.46:0/2828155848 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.559+0000 7f10c7fff640 1 -- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8070940 msgr2=0x7f10c809b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.559+0000 7f10c7fff640 1 --2- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8070940 0x7f10c809b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.559+0000 7f10c7fff640 1 -- 172.21.15.46:0/2828155848 >> v1:172.21.15.78:6789/0 conn(0x7f10c8160310 legacy=0x7f10c8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.559+0000 7f10c7fff640 1 -- 172.21.15.46:0/2828155848 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c815fce0 con 0x7f10c80d0da0 2024-04-15T15:17:39.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.559+0000 7f10c67fc640 1 -- 172.21.15.46:0/2828155848 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702915063 0 0) 0x7f10c80d05e0 con 0x7f10c8160310 2024-04-15T15:17:39.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10c7fff640 1 --2- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80d1230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009d90 tx=0x7f10c002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c54f98d1c438bbf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:39.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10c67fc640 1 -- 172.21.15.46:0/2828155848 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0039070 con 0x7f10c80d0da0 2024-04-15T15:17:39.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10c67fc640 1 -- 172.21.15.46:0/2828155848 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c002cc50 con 0x7f10c80d0da0 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10c67fc640 1 -- 172.21.15.46:0/2828155848 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c0034440 con 0x7f10c80d0da0 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 -- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 msgr2=0x7f10c80d1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80d1230 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009d90 tx=0x7f10c002bea0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 -- 172.21.15.46:0/2828155848 shutdown_connections 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80d1230 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2828155848 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8070940 0x7f10c809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 -- 172.21.15.46:0/2828155848 >> 172.21.15.46:0/2828155848 conn(0x7f10c80b3f40 msgr2=0x7f10c80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 -- 172.21.15.46:0/2828155848 shutdown_connections 2024-04-15T15:17:39.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.560+0000 7f10cdb32640 1 -- 172.21.15.46:0/2828155848 wait complete. 2024-04-15T15:17:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 Processor -- start 2024-04-15T15:17:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 -- start start 2024-04-15T15:17:39.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 0x7f10c80b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8160310 0x7f10c80b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f10c80d1770 con 0x7f10c8070940 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f10c8160770 con 0x7f10c80d0da0 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.562+0000 7f10cdb32640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f10c80d18f0 con 0x7f10c8160310 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 0x7f10c80b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 0x7f10c80b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40654/0 (socket says 172.21.15.46:40654) 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 -- 172.21.15.46:0/2554977686 learned_addr learned my addr 172.21.15.46:0/2554977686 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c7fff640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8160310 0x7f10c80b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c6ffd640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 -- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8160310 msgr2=0x7f10c80b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8160310 0x7f10c80b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 -- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 msgr2=0x7f10c80b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 -- 172.21.15.46:0/2554977686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c80b6dd0 con 0x7f10c8070940 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10c77fe640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 0x7f10c80b76f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f10b800e870 tx=0x7f10b800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8017070 con 0x7f10c8070940 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b800c940 con 0x7f10c8070940 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c81f52d0 con 0x7f10c8070940 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b800cd30 con 0x7f10c8070940 2024-04-15T15:17:39.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.563+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c81f8240 con 0x7f10c8070940 2024-04-15T15:17:39.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.564+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c80689e0 con 0x7f10c8070940 2024-04-15T15:17:39.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.565+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f10b801a070 con 0x7f10c8070940 2024-04-15T15:17:39.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.565+0000 7f10affff640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f10b406c980 0x7f10b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:39.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.565+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f10b808bd30 con 0x7f10c8070940 2024-04-15T15:17:39.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.565+0000 7f10c6ffd640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f10b406c980 0x7f10b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:39.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.566+0000 7f10c6ffd640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f10b406c980 0x7f10b406ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f10bc000f60 tx=0x7f10bc007b50 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:39.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.569+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10b8059ef0 con 0x7f10c8070940 2024-04-15T15:17:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.766+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f10c8070720 con 0x7f10c8070940 2024-04-15T15:17:39.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.766+0000 7f10affff640 1 -- 172.21.15.46:0/2554977686 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+120 (secure 0 0 0) 0x7f10b80597f0 con 0x7f10c8070940 2024-04-15T15:17:39.768 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:39.769 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:39.769 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f10b406c980 msgr2=0x7f10b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f10b406c980 0x7f10b406ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f10bc000f60 tx=0x7f10bc007b50 comp rx=0 tx=0).stop 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 msgr2=0x7f10c80b76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 0x7f10c80b76f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f10b800e870 tx=0x7f10b800ec70 comp rx=0 tx=0).stop 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 shutdown_connections 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f10b406c980 0x7f10b406ee40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f10c8160310 0x7f10c80b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10c80d0da0 0x7f10c80b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 --2- 172.21.15.46:0/2554977686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10c8070940 0x7f10c80b76f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 >> 172.21.15.46:0/2554977686 conn(0x7f10c80b3f40 msgr2=0x7f10c8070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 shutdown_connections 2024-04-15T15:17:39.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:39.768+0000 7f10cdb32640 1 -- 172.21.15.46:0/2554977686 wait complete. 2024-04-15T15:17:39.781 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-04-15T15:17:39.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:39.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:39.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi078.stderr:ceph-fuse[45245]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:39.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:17:39.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:39.816 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:17:39.991 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:39.991 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:39.991 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:40.009 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:40.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:40.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.154+0000 7f987bf1b640 1 Processor -- start 2024-04-15T15:17:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.154+0000 7f987bf1b640 1 -- start start 2024-04-15T15:17:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987bf1b640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f98740af5e0 0x7f98740ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987bf1b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98740b00d0 0x7f98741682c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987bf1b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987406ed20 con 0x7f987406e560 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987bf1b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f98740b6140 con 0x7f98740b00d0 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987bf1b640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f987406f000 con 0x7f98740af5e0 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98740b00d0 0x7f98741682c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f98740b00d0 0x7f98741682c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40656/0 (socket says 172.21.15.46:40656) 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 -- 172.21.15.46:0/3754603333 learned_addr learned my addr 172.21.15.46:0/3754603333 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 -- 172.21.15.46:0/3754603333 >> v2:172.21.15.78:3300/0 conn(0x7f98740af5e0 msgr2=0x7f98740ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f9879cb7640 1 --2- 172.21.15.46:0/3754603333 >> v2:172.21.15.78:3300/0 conn(0x7f98740af5e0 0x7f98740ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 --2- 172.21.15.46:0/3754603333 >> v2:172.21.15.78:3300/0 conn(0x7f98740af5e0 0x7f98740ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 -- 172.21.15.46:0/3754603333 >> v1:172.21.15.78:6789/0 conn(0x7f987406e560 legacy=0x7f98740af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.155+0000 7f987a4b8640 1 -- 172.21.15.46:0/3754603333 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987406f2e0 con 0x7f98740b00d0 2024-04-15T15:17:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.156+0000 7f987a4b8640 1 --2- 172.21.15.46:0/3754603333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741682c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f98740677a0 tx=0x7f986c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6e4a4ab3e74e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.156+0000 7f9878cb5640 1 -- 172.21.15.46:0/3754603333 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c02ca80 con 0x7f98740b00d0 2024-04-15T15:17:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.156+0000 7f9878cb5640 1 -- 172.21.15.46:0/3754603333 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986c02cc20 con 0x7f98740b00d0 2024-04-15T15:17:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.156+0000 7f9878cb5640 1 -- 172.21.15.46:0/3754603333 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c0343f0 con 0x7f98740b00d0 2024-04-15T15:17:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 -- 172.21.15.46:0/3754603333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 msgr2=0x7f98741682c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 --2- 172.21.15.46:0/3754603333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741682c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f98740677a0 tx=0x7f986c02bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 -- 172.21.15.46:0/3754603333 shutdown_connections 2024-04-15T15:17:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 --2- 172.21.15.46:0/3754603333 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741682c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 --2- 172.21.15.46:0/3754603333 >> v2:172.21.15.78:3300/0 conn(0x7f98740af5e0 0x7f98740ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 -- 172.21.15.46:0/3754603333 >> 172.21.15.46:0/3754603333 conn(0x7f987406b9e0 msgr2=0x7f987406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 -- 172.21.15.46:0/3754603333 shutdown_connections 2024-04-15T15:17:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.157+0000 7f987bf1b640 1 -- 172.21.15.46:0/3754603333 wait complete. 2024-04-15T15:17:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.158+0000 7f987bf1b640 1 Processor -- start 2024-04-15T15:17:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.158+0000 7f987bf1b640 1 -- start start 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f987406e560 0x7f9874157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98740af5e0 0x7f9874160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741f4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f987406e9d0 con 0x7f98740b00d0 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f987406f8a0 con 0x7f98740af5e0 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f987415ea40 con 0x7f987406e560 2024-04-15T15:17:40.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741f4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741f4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40664/0 (socket says 172.21.15.46:40664) 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 -- 172.21.15.46:0/4096810402 learned_addr learned my addr 172.21.15.46:0/4096810402 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f98794b6640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98740af5e0 0x7f9874160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f9879cb7640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f987406e560 0x7f9874157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 -- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f987406e560 msgr2=0x7f9874157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f987406e560 0x7f9874157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 -- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98740af5e0 msgr2=0x7f9874160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98740af5e0 0x7f9874160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 -- 172.21.15.46:0/4096810402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98741f4370 con 0x7f98740b00d0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f98794b6640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98740af5e0 0x7f9874160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987a4b8640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741f4b50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f986c02c4f0 tx=0x7f986c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c043020 con 0x7f98740b00d0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986c03e450 con 0x7f98740b00d0 2024-04-15T15:17:40.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.159+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98741617b0 con 0x7f98740b00d0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.160+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c03e820 con 0x7f98740b00d0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.160+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987414f7c0 con 0x7f98740b00d0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.161+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98740690f0 con 0x7f98740b00d0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.161+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f986c046020 con 0x7f98740b00d0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.161+0000 7f986a7fc640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f985806c980 0x7f985806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.161+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f986c0aea50 con 0x7f98740b00d0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.162+0000 7f9879cb7640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f985806c980 0x7f985806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.162+0000 7f9879cb7640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f985806c980 0x7f985806ee40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f987000a970 tx=0x7f9870008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.166+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f986c04d020 con 0x7f98740b00d0 2024-04-15T15:17:40.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.350+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f987415b460 con 0x7f98740b00d0 2024-04-15T15:17:40.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.351+0000 7f986a7fc640 1 -- 172.21.15.46:0/4096810402 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7f986c07ce40 con 0x7f98740b00d0 2024-04-15T15:17:40.353 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:40.353 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.353+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f985806c980 msgr2=0x7f985806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.353+0000 7f987bf1b640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f985806c980 0x7f985806ee40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f987000a970 tx=0x7f9870008040 comp rx=0 tx=0).stop 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.353+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 msgr2=0x7f98741f4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.353+0000 7f987bf1b640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741f4b50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f986c02c4f0 tx=0x7f986c004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 shutdown_connections 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f985806c980 0x7f985806ee40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98740b00d0 0x7f98741f4b50 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98740af5e0 0x7f9874160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 --2- 172.21.15.46:0/4096810402 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f987406e560 0x7f9874157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 >> 172.21.15.46:0/4096810402 conn(0x7f987406b9e0 msgr2=0x7f98740af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 shutdown_connections 2024-04-15T15:17:40.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.354+0000 7f987bf1b640 1 -- 172.21.15.46:0/4096810402 wait complete. 2024-04-15T15:17:40.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:40.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.512+0000 7fc7c5f3d640 1 Processor -- start 2024-04-15T15:17:40.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7c5f3d640 1 -- start start 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7c5f3d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7c00d2360 0x7fc7c00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7c5f3d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7c00b47a0 con 0x7fc7c015e700 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7c5f3d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7c00b4a80 con 0x7fc7c015eae0 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7c5f3d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc7c00b53b0 con 0x7fc7c00d2360 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7beffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc7c00d2360 0x7fc7c00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7bf7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc7c015e700 0x7fc7c00b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51970/0 (socket says 172.21.15.46:51970) 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.513+0000 7fc7bf7fe640 1 -- 172.21.15.46:0/1749727094 learned_addr learned my addr 172.21.15.46:0/1749727094 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.514+0000 7fc7beffd640 1 -- 172.21.15.46:0/1749727094 >> v1:172.21.15.78:6789/0 conn(0x7fc7c015eae0 legacy=0x7fc7c00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.514+0000 7fc7beffd640 1 -- 172.21.15.46:0/1749727094 >> v1:172.21.15.46:6789/0 conn(0x7fc7c015e700 legacy=0x7fc7c00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.514+0000 7fc7be7fc640 1 -- 172.21.15.46:0/1749727094 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277938494 0 0) 0x7fc7c00b47a0 con 0x7fc7c015e700 2024-04-15T15:17:40.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.514+0000 7fc7beffd640 1 -- 172.21.15.46:0/1749727094 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c00b4d60 con 0x7fc7c00d2360 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7beffd640 1 --2- 172.21.15.46:0/1749727094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c00d2360 0x7fc7c00d2b60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00b5530 tx=0x7fc7b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899803dc8fed4beb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7be7fc640 1 -- 172.21.15.46:0/1749727094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b4034070 con 0x7fc7c00d2360 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7be7fc640 1 -- 172.21.15.46:0/1749727094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b4030610 con 0x7fc7c00d2360 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7be7fc640 1 -- 172.21.15.46:0/1749727094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b40309e0 con 0x7fc7c00d2360 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/1749727094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c00d2360 msgr2=0x7fc7c00d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/1749727094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c00d2360 0x7fc7c00d2b60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00b5530 tx=0x7fc7b402b960 comp rx=0 tx=0).stop 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/1749727094 shutdown_connections 2024-04-15T15:17:40.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/1749727094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c00d2360 0x7fc7c00d2b60 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/1749727094 >> 172.21.15.46:0/1749727094 conn(0x7fc7c008bf60 msgr2=0x7fc7c00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/1749727094 shutdown_connections 2024-04-15T15:17:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.515+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/1749727094 wait complete. 2024-04-15T15:17:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.516+0000 7fc7c5f3d640 1 Processor -- start 2024-04-15T15:17:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 -- start start 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7c00d2360 0x7fc7c0153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc7c015e700 0x7fc7c0160110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 0x7fc7c00c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c00d1d40 con 0x7fc7c015eae0 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c00b4450 con 0x7fc7c00d2360 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7c5f3d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc7c00d1ec0 con 0x7fc7c015e700 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 0x7fc7c00c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 0x7fc7c00c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40680/0 (socket says 172.21.15.46:40680) 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7bffff640 1 -- 172.21.15.46:0/2370904725 learned_addr learned my addr 172.21.15.46:0/2370904725 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.517+0000 7fc7bf7fe640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7c00d2360 0x7fc7c0153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7beffd640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc7c015e700 0x7fc7c0160110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7bffff640 1 -- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc7c015e700 msgr2=0x7fc7c0160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7bffff640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc7c015e700 0x7fc7c0160110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7bffff640 1 -- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7c00d2360 msgr2=0x7fc7c0153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7bffff640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7c00d2360 0x7fc7c0153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7bffff640 1 -- 172.21.15.46:0/2370904725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c0154300 con 0x7fc7c015eae0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7bffff640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 0x7fc7c00c7f70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc7b800c3c0 tx=0x7fc7b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8017070 con 0x7fc7c015eae0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b80052d0 con 0x7fc7c015eae0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c015f070 con 0x7fc7c015eae0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8003810 con 0x7fc7c015eae0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.518+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c015f630 con 0x7fc7c015eae0 2024-04-15T15:17:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.519+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c00689e0 con 0x7fc7c015eae0 2024-04-15T15:17:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.520+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc7b801a070 con 0x7fc7c015eae0 2024-04-15T15:17:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.520+0000 7fc7a7fff640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc7ac06c9b0 0x7fc7ac06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.520+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc7b808b080 con 0x7fc7c015eae0 2024-04-15T15:17:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.520+0000 7fc7bf7fe640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc7ac06c9b0 0x7fc7ac06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.521+0000 7fc7bf7fe640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc7ac06c9b0 0x7fc7ac06ee70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00d0950 tx=0x7fc7b0009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.524+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7b80592d0 con 0x7fc7c015eae0 2024-04-15T15:17:40.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.709+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc7c00ba500 con 0x7fc7c015eae0 2024-04-15T15:17:40.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.710+0000 7fc7a7fff640 1 -- 172.21.15.46:0/2370904725 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fc7b8058bd0 con 0x7fc7c015eae0 2024-04-15T15:17:40.712 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc7ac06c9b0 msgr2=0x7fc7ac06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc7ac06c9b0 0x7fc7ac06ee70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00d0950 tx=0x7fc7b0009040 comp rx=0 tx=0).stop 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 msgr2=0x7fc7c00c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 0x7fc7c00c7f70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc7b800c3c0 tx=0x7fc7b800c880 comp rx=0 tx=0).stop 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 shutdown_connections 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc7ac06c9b0 0x7fc7ac06ee70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7c015eae0 0x7fc7c00c7f70 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc7c015e700 0x7fc7c0160110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 --2- 172.21.15.46:0/2370904725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7c00d2360 0x7fc7c0153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 >> 172.21.15.46:0/2370904725 conn(0x7fc7c008bf60 msgr2=0x7fc7c006e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 shutdown_connections 2024-04-15T15:17:40.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.712+0000 7fc7c5f3d640 1 -- 172.21.15.46:0/2370904725 wait complete. 2024-04-15T15:17:40.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:40.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.830+0000 7f2a9c60c640 1 Processor -- start 2024-04-15T15:17:40.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.831+0000 7f2a9c60c640 1 -- start start 2024-04-15T15:17:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.831+0000 7f2a9c60c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2a9415e700 0x7f2a940b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.831+0000 7f2a9c60c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a940d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9c60c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a940b47a0 con 0x7f2a9415eae0 2024-04-15T15:17:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9c60c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2a940b53b0 con 0x7f2a9415e700 2024-04-15T15:17:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9c60c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2a940b4a80 con 0x7f2a940d2360 2024-04-15T15:17:40.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a99ba7640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a940d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2a9415e700 0x7f2a940b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2a9415e700 0x7f2a940b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:49864/0 (socket says 172.21.15.46:49864) 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 -- 172.21.15.46:0/3328438505 learned_addr learned my addr 172.21.15.46:0/3328438505 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a99ba7640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a940d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:57208/0 (socket says 172.21.15.46:57208) 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 -- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 msgr2=0x7f2a940d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 --2- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a940d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 -- 172.21.15.46:0/3328438505 >> v1:172.21.15.78:6789/0 conn(0x7f2a9415eae0 legacy=0x7f2a940cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a9a3a8640 1 -- 172.21.15.46:0/3328438505 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a940b4d60 con 0x7f2a9415e700 2024-04-15T15:17:40.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.832+0000 7f2a99ba7640 1 --2- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a940d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:40.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.833+0000 7f2a9a3a8640 1 --2- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b5840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009850 tx=0x7f2a9002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aedea92e7cf545b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.833+0000 7f2a993a6640 1 -- 172.21.15.46:0/3328438505 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90004230 con 0x7f2a9415e700 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.833+0000 7f2a993a6640 1 -- 172.21.15.46:0/3328438505 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a900043d0 con 0x7f2a9415e700 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.833+0000 7f2a993a6640 1 -- 172.21.15.46:0/3328438505 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a90003810 con 0x7f2a9415e700 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 -- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 msgr2=0x7f2a940b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b5840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2a90009850 tx=0x7f2a9002b960 comp rx=0 tx=0).stop 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 -- 172.21.15.46:0/3328438505 shutdown_connections 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a940d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/3328438505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b5840 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 -- 172.21.15.46:0/3328438505 >> 172.21.15.46:0/3328438505 conn(0x7f2a9408bf60 msgr2=0x7f2a940d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 -- 172.21.15.46:0/3328438505 shutdown_connections 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.834+0000 7f2a9c60c640 1 -- 172.21.15.46:0/3328438505 wait complete. 2024-04-15T15:17:40.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.835+0000 7f2a9c60c640 1 Processor -- start 2024-04-15T15:17:40.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.835+0000 7f2a9c60c640 1 -- start start 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9c60c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a9414dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9c60c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9c60c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 0x7f2a940cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9c60c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940b5d80 con 0x7f2a9415eae0 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9c60c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940b4450 con 0x7f2a9415e700 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9c60c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2a940d1d40 con 0x7f2a940d2360 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 0x7f2a940cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 0x7f2a940cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40684/0 (socket says 172.21.15.46:40684) 2024-04-15T15:17:40.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 -- 172.21.15.46:0/1476700939 learned_addr learned my addr 172.21.15.46:0/1476700939 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9a3a8640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a9414dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a99ba7640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 -- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 msgr2=0x7f2a9414dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a9414dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 -- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 msgr2=0x7f2a940b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.836+0000 7f2a9aba9640 1 -- 172.21.15.46:0/1476700939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a940b79d0 con 0x7f2a9415eae0 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.837+0000 7f2a9aba9640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 0x7f2a940cd1b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c6b0 tx=0x7f2a8c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.837+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c003b70 con 0x7f2a9415eae0 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.837+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8c003d30 con 0x7f2a9415eae0 2024-04-15T15:17:40.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.837+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c005310 con 0x7f2a9415eae0 2024-04-15T15:17:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.837+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a940b7cb0 con 0x7f2a9415eae0 2024-04-15T15:17:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.837+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a940b8270 con 0x7f2a9415eae0 2024-04-15T15:17:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.838+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a940689e0 con 0x7f2a9415eae0 2024-04-15T15:17:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.838+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2a8c0054b0 con 0x7f2a9415eae0 2024-04-15T15:17:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.839+0000 7f2a8affd640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2a7806c980 0x7f2a7806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:40.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.839+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f2a8c08bb40 con 0x7f2a9415eae0 2024-04-15T15:17:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.839+0000 7f2a9a3a8640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2a7806c980 0x7f2a7806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:40.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.839+0000 7f2a9a3a8640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2a7806c980 0x7f2a7806ee40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2a90035000 tx=0x7f2a90008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:40.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:40.843+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a8c0596b0 con 0x7f2a9415eae0 2024-04-15T15:17:41.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.027+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2a940ba500 con 0x7f2a9415eae0 2024-04-15T15:17:41.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.027+0000 7f2a8affd640 1 -- 172.21.15.46:0/1476700939 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f2a8c05d9b0 con 0x7f2a9415eae0 2024-04-15T15:17:41.029 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2a7806c980 msgr2=0x7f2a7806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2a7806c980 0x7f2a7806ee40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2a90035000 tx=0x7f2a90008030 comp rx=0 tx=0).stop 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 msgr2=0x7f2a940cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 0x7f2a940cd1b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c00c6b0 tx=0x7f2a8c00cb70 comp rx=0 tx=0).stop 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 shutdown_connections 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2a7806c980 0x7f2a7806ee40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a9415eae0 0x7f2a940cd1b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2a9415e700 0x7f2a940b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 --2- 172.21.15.46:0/1476700939 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2a940d2360 0x7f2a9414dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 >> 172.21.15.46:0/1476700939 conn(0x7f2a9408bf60 msgr2=0x7f2a9415ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 shutdown_connections 2024-04-15T15:17:41.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.029+0000 7f2a9c60c640 1 -- 172.21.15.46:0/1476700939 wait complete. 2024-04-15T15:17:41.042 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.150+0000 7f40aa1e2640 1 Processor -- start 2024-04-15T15:17:41.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.150+0000 7f40aa1e2640 1 -- start start 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40aa1e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40a415ea20 0x7f40a406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40aa1e2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a415ee00 con 0x7f40a40d0650 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40aa1e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a40b24f0 con 0x7f40a40b1ae0 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40aa1e2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f40a40b22c0 con 0x7f40a415ea20 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40a3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f40a40b1ae0 0x7f40a40cdb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:40060/0 (socket says 172.21.15.46:40060) 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40a37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f40a415ea20 0x7f40a406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.151+0000 7f40a3fff640 1 -- 172.21.15.46:0/648585740 learned_addr learned my addr 172.21.15.46:0/648585740 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.152+0000 7f40a37fe640 1 -- 172.21.15.46:0/648585740 >> v1:172.21.15.78:6789/0 conn(0x7f40a40b1ae0 legacy=0x7f40a40cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.152+0000 7f40a37fe640 1 -- 172.21.15.46:0/648585740 >> v1:172.21.15.46:6789/0 conn(0x7f40a40d0650 legacy=0x7f40a40d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.152+0000 7f40a37fe640 1 -- 172.21.15.46:0/648585740 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a40b27d0 con 0x7f40a415ea20 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.152+0000 7f40a27fc640 1 -- 172.21.15.46:0/648585740 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2524637798 0 0) 0x7f40a415ee00 con 0x7f40a40d0650 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.152+0000 7f40a37fe640 1 --2- 172.21.15.46:0/648585740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a406c3f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f40940097a0 tx=0x7f409402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c4d50d99edd222 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40a27fc640 1 -- 172.21.15.46:0/648585740 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094035020 con 0x7f40a415ea20 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40a27fc640 1 -- 172.21.15.46:0/648585740 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094031830 con 0x7f40a415ea20 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40a27fc640 1 -- 172.21.15.46:0/648585740 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094031c00 con 0x7f40a415ea20 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 -- 172.21.15.46:0/648585740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 msgr2=0x7f40a406c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/648585740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a406c3f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f40940097a0 tx=0x7f409402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 -- 172.21.15.46:0/648585740 shutdown_connections 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/648585740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a406c3f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 -- 172.21.15.46:0/648585740 >> 172.21.15.46:0/648585740 conn(0x7f40a409a9d0 msgr2=0x7f40a40b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:41.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 -- 172.21.15.46:0/648585740 shutdown_connections 2024-04-15T15:17:41.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.153+0000 7f40aa1e2640 1 -- 172.21.15.46:0/648585740 wait complete. 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.154+0000 7f40aa1e2640 1 Processor -- start 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 -- start start 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40a40b1ae0 0x7f40a415c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40a40d0650 0x7f40a40b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a40b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40b2d90 con 0x7f40a415ea20 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f40a40b0640 con 0x7f40a40d0650 2024-04-15T15:17:41.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40aa1e2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f40a40ce210 con 0x7f40a40b1ae0 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a40b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a40b4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40702/0 (socket says 172.21.15.46:40702) 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.155+0000 7f40a3fff640 1 -- 172.21.15.46:0/3671604801 learned_addr learned my addr 172.21.15.46:0/3671604801 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a37fe640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40a40b1ae0 0x7f40a415c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a2ffd640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40a40d0650 0x7f40a40b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a3fff640 1 -- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40a40b1ae0 msgr2=0x7f40a415c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a3fff640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40a40b1ae0 0x7f40a415c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a3fff640 1 -- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40a40d0650 msgr2=0x7f40a40b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a3fff640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40a40d0650 0x7f40a40b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a3fff640 1 -- 172.21.15.46:0/3671604801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a415c730 con 0x7f40a415ea20 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a37fe640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40a40b1ae0 0x7f40a415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a2ffd640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40a40d0650 0x7f40a40b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:41.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40a3fff640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a40b4e90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f409c00cbc0 tx=0x7f409c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c017070 con 0x7f40a415ea20 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409c00ad40 con 0x7f40a415ea20 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a40cb0e0 con 0x7f40a415ea20 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409c012600 con 0x7f40a415ea20 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.156+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a4160430 con 0x7f40a415ea20 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.157+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a40690f0 con 0x7f40a415ea20 2024-04-15T15:17:41.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.158+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f409c01a070 con 0x7f40a415ea20 2024-04-15T15:17:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.158+0000 7f408bfff640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f409006c980 0x7f409006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.158+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f409c08ba30 con 0x7f40a415ea20 2024-04-15T15:17:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.158+0000 7f40a37fe640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f409006c980 0x7f409006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.159+0000 7f40a37fe640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f409006c980 0x7f409006ee40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4094036000 tx=0x7f409403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:41.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.162+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f409c059620 con 0x7f40a415ea20 2024-04-15T15:17:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.348+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f40a40b3b30 con 0x7f40a415ea20 2024-04-15T15:17:41.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.348+0000 7f408bfff640 1 -- 172.21.15.46:0/3671604801 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f40a40b3b30 con 0x7f40a415ea20 2024-04-15T15:17:41.350 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f409006c980 msgr2=0x7f409006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f409006c980 0x7f409006ee40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4094036000 tx=0x7f409403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 msgr2=0x7f40a40b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a40b4e90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f409c00cbc0 tx=0x7f409c00a430 comp rx=0 tx=0).stop 2024-04-15T15:17:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 shutdown_connections 2024-04-15T15:17:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f409006c980 0x7f409006ee40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40a415ea20 0x7f40a40b4e90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40a40d0650 0x7f40a40b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 --2- 172.21.15.46:0/3671604801 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40a40b1ae0 0x7f40a415c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 >> 172.21.15.46:0/3671604801 conn(0x7f40a409a9d0 msgr2=0x7f40a40cc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 shutdown_connections 2024-04-15T15:17:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.350+0000 7f40aa1e2640 1 -- 172.21.15.46:0/3671604801 wait complete. 2024-04-15T15:17:41.363 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.467+0000 7fec67669640 1 Processor -- start 2024-04-15T15:17:41.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.467+0000 7fec67669640 1 -- start start 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec67669640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fec601602d0 0x7fec600cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec67669640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fec6009ab40 0x7fec6009afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec67669640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6015dbd0 con 0x7fec601606b0 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec67669640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fec600cd6a0 con 0x7fec6009ab40 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec67669640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fec600cd980 con 0x7fec601602d0 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fec6009ab40 0x7fec6009afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fec6009ab40 0x7fec6009afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40714/0 (socket says 172.21.15.46:40714) 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 -- 172.21.15.46:0/203180847 learned_addr learned my addr 172.21.15.46:0/203180847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65405640 1 --2- 172.21.15.46:0/203180847 >> v2:172.21.15.78:3300/0 conn(0x7fec601602d0 0x7fec600cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 -- 172.21.15.46:0/203180847 >> v2:172.21.15.78:3300/0 conn(0x7fec601602d0 msgr2=0x7fec600cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 --2- 172.21.15.46:0/203180847 >> v2:172.21.15.78:3300/0 conn(0x7fec601602d0 0x7fec600cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 -- 172.21.15.46:0/203180847 >> v1:172.21.15.78:6789/0 conn(0x7fec601606b0 legacy=0x7fec600d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65c06640 1 -- 172.21.15.46:0/203180847 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec600cdc60 con 0x7fec6009ab40 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.468+0000 7fec65405640 1 --2- 172.21.15.46:0/203180847 >> v2:172.21.15.78:3300/0 conn(0x7fec601602d0 0x7fec600cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.469+0000 7fec65c06640 1 --2- 172.21.15.46:0/203180847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec6009ab40 0x7fec6009afd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fec600734f0 tx=0x7fec5802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6be62a37f8a79e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.469+0000 7fec57fff640 1 -- 172.21.15.46:0/203180847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5802ca80 con 0x7fec6009ab40 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.469+0000 7fec57fff640 1 -- 172.21.15.46:0/203180847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec5802cc20 con 0x7fec6009ab40 2024-04-15T15:17:41.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.469+0000 7fec57fff640 1 -- 172.21.15.46:0/203180847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec580343f0 con 0x7fec6009ab40 2024-04-15T15:17:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 -- 172.21.15.46:0/203180847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec6009ab40 msgr2=0x7fec6009afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 --2- 172.21.15.46:0/203180847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec6009ab40 0x7fec6009afd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fec600734f0 tx=0x7fec5802bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 -- 172.21.15.46:0/203180847 shutdown_connections 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 --2- 172.21.15.46:0/203180847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec6009ab40 0x7fec6009afd0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 --2- 172.21.15.46:0/203180847 >> v2:172.21.15.78:3300/0 conn(0x7fec601602d0 0x7fec600cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 -- 172.21.15.46:0/203180847 >> 172.21.15.46:0/203180847 conn(0x7fec600d2460 msgr2=0x7fec600d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 -- 172.21.15.46:0/203180847 shutdown_connections 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.470+0000 7fec67669640 1 -- 172.21.15.46:0/203180847 wait complete. 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.471+0000 7fec67669640 1 Processor -- start 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.471+0000 7fec67669640 1 -- start start 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec67669640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec6009ab40 0x7fec600b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec67669640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 0x7fec60150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec67669640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fec601606b0 0x7fec601f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec67669640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fec60160b10 con 0x7fec601602d0 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec67669640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fec600ce220 con 0x7fec6009ab40 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec67669640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fec6009b510 con 0x7fec601606b0 2024-04-15T15:17:41.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 0x7fec60150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 0x7fec60150850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40722/0 (socket says 172.21.15.46:40722) 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 -- 172.21.15.46:0/3876726671 learned_addr learned my addr 172.21.15.46:0/3876726671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec65405640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec6009ab40 0x7fec600b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec65c06640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fec601606b0 0x7fec601f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 -- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fec601606b0 msgr2=0x7fec601f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fec601606b0 0x7fec601f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 -- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec6009ab40 msgr2=0x7fec600b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec6009ab40 0x7fec600b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 -- 172.21.15.46:0/3876726671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec600b1070 con 0x7fec601602d0 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec65405640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec6009ab40 0x7fec600b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec65c06640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fec601606b0 0x7fec601f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:41.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.472+0000 7fec64c04640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 0x7fec60150850 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fec5000b420 tx=0x7fec5000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.473+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50011830 con 0x7fec601602d0 2024-04-15T15:17:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.473+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec50016070 con 0x7fec601602d0 2024-04-15T15:17:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.473+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec600cac20 con 0x7fec601602d0 2024-04-15T15:17:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.473+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50011c10 con 0x7fec601602d0 2024-04-15T15:17:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.473+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec600cb1e0 con 0x7fec601602d0 2024-04-15T15:17:41.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.474+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec600689e0 con 0x7fec601602d0 2024-04-15T15:17:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.474+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fec5001b460 con 0x7fec601602d0 2024-04-15T15:17:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.475+0000 7fec55ffb640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fec4406c980 0x7fec4406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.475+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fec5008b830 con 0x7fec601602d0 2024-04-15T15:17:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.475+0000 7fec65405640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fec4406c980 0x7fec4406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:41.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.476+0000 7fec65405640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fec4406c980 0x7fec4406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec600cfae0 tx=0x7fec5c008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:41.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.479+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec50020020 con 0x7fec601602d0 2024-04-15T15:17:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.679+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fec600b4e20 con 0x7fec601602d0 2024-04-15T15:17:41.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.679+0000 7fec55ffb640 1 -- 172.21.15.46:0/3876726671 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+120 (secure 0 0 0) 0x7fec50059a70 con 0x7fec601602d0 2024-04-15T15:17:41.681 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:41.681 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:41.681 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fec4406c980 msgr2=0x7fec4406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fec4406c980 0x7fec4406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fec600cfae0 tx=0x7fec5c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 msgr2=0x7fec60150850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 0x7fec60150850 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fec5000b420 tx=0x7fec5000b8e0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 shutdown_connections 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fec4406c980 0x7fec4406ee40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fec601606b0 0x7fec601f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec601602d0 0x7fec60150850 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 --2- 172.21.15.46:0/3876726671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec6009ab40 0x7fec600b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:41.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 >> 172.21.15.46:0/3876726671 conn(0x7fec600d2460 msgr2=0x7fec600b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 shutdown_connections 2024-04-15T15:17:41.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:41.681+0000 7fec67669640 1 -- 172.21.15.46:0/3876726671 wait complete. 2024-04-15T15:17:41.694 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-04-15T15:17:41.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:41.694 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:17:41.721 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[56029]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:41.722 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:17:41.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:41.723 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:17:41.888 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:41.889 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:41.889 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:41.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:42.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.031+0000 7f2bd0100640 1 Processor -- start 2024-04-15T15:17:42.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.032+0000 7f2bd0100640 1 -- start start 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.032+0000 7f2bd0100640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f2bc815e790 0x7f2bc80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.032+0000 7f2bd0100640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 0x7f2bc80d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.032+0000 7f2bd0100640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bc80d2480 con 0x7f2bc80cfd20 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.032+0000 7f2bd0100640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f2bc80d2db0 con 0x7f2bc815e790 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.032+0000 7f2bd0100640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc80d2760 con 0x7f2bc80d1610 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcd69b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2bc80cfd20 0x7f2bc80d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51984/0 (socket says 172.21.15.46:51984) 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcd69b640 1 -- 172.21.15.46:0/4187698249 learned_addr learned my addr 172.21.15.46:0/4187698249 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:42.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcde9c640 1 --2- 172.21.15.46:0/4187698249 >> v2:172.21.15.78:3300/0 conn(0x7f2bc815e790 0x7f2bc80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bce69d640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 0x7f2bc80d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136987180 0 0) 0x7f2bc80d2480 con 0x7f2bc80cfd20 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bb4003520 con 0x7f2bc80cfd20 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3188715505 0 0) 0x7f2bb4003520 con 0x7f2bc80cfd20 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 msgr2=0x7f2bc80d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 0x7f2bc80d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 >> v2:172.21.15.78:3300/0 conn(0x7f2bc815e790 msgr2=0x7f2bc80d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 --2- 172.21.15.46:0/4187698249 >> v2:172.21.15.78:3300/0 conn(0x7f2bc815e790 0x7f2bc80d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc80d2a40 con 0x7f2bc80cfd20 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f2bb8001670 con 0x7f2bc80cfd20 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 >> v1:172.21.15.46:6789/0 conn(0x7f2bc80cfd20 legacy=0x7f2bc80d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40039b0 0x7f2bb4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4004800 0x7f2bb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 0x7f2bb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc80d2db0 con 0x7f2bb4004800 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc80d2760 con 0x7f2bb40039b0 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb400b9c0 con 0x7f2bc80d1610 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4004800 0x7f2bb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bcde9c640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40039b0 0x7f2bb4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.033+0000 7f2bce69d640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 0x7f2bb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.034+0000 7f2bcd69b640 1 -- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 msgr2=0x7f2bb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.034+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80d1610 0x7f2bb40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.034+0000 7f2bcd69b640 1 -- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40039b0 msgr2=0x7f2bb4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.034+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb40039b0 0x7f2bb4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.034+0000 7f2bcd69b640 1 -- 172.21.15.46:0/4187698249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb4003520 con 0x7f2bb4004800 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.034+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4004800 0x7f2bb4004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800c240 tx=0x7f2bb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5348ad2247a1aa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8007d30 con 0x7f2bb4004800 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb8030070 con 0x7f2bb4004800 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bcce9a640 1 -- 172.21.15.46:0/4187698249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb802cd60 con 0x7f2bb4004800 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bd0100640 1 -- 172.21.15.46:0/4187698249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4004800 msgr2=0x7f2bb4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bd0100640 1 --2- 172.21.15.46:0/4187698249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bb4004800 0x7f2bb4004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800c240 tx=0x7f2bb4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bce69d640 1 -- 172.21.15.46:0/4187698249 reap_dead start 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bd0100640 1 -- 172.21.15.46:0/4187698249 shutdown_connections 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bd0100640 1 -- 172.21.15.46:0/4187698249 >> 172.21.15.46:0/4187698249 conn(0x7f2bc808bf60 msgr2=0x7f2bc80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bd0100640 1 -- 172.21.15.46:0/4187698249 shutdown_connections 2024-04-15T15:17:42.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.035+0000 7f2bd0100640 1 -- 172.21.15.46:0/4187698249 wait complete. 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.036+0000 7f2bd0100640 1 Processor -- start 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.036+0000 7f2bd0100640 1 -- start start 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bd0100640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb4004800 0x7f2bc81f83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bd0100640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 0x7f2bc814e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bd0100640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80c9ac0 0x7f2bc80c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bd0100640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc815ecf0 con 0x7f2bc814e260 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bd0100640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc815eb70 con 0x7f2bb4004800 2024-04-15T15:17:42.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bd0100640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc80d0160 con 0x7f2bc80c9ac0 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bcd69b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 0x7f2bc814e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bcd69b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 0x7f2bc814e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40748/0 (socket says 172.21.15.46:40748) 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.037+0000 7f2bcd69b640 1 -- 172.21.15.46:0/523985537 learned_addr learned my addr 172.21.15.46:0/523985537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcd69b640 1 -- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80c9ac0 msgr2=0x7f2bc80c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bce69d640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80c9ac0 0x7f2bc80c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcde9c640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb4004800 0x7f2bc81f83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80c9ac0 0x7f2bc80c9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcd69b640 1 -- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb4004800 msgr2=0x7f2bc81f83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb4004800 0x7f2bc81f83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcd69b640 1 -- 172.21.15.46:0/523985537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc81f88e0 con 0x7f2bc814e260 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bce69d640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80c9ac0 0x7f2bc80c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcde9c640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb4004800 0x7f2bc81f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bcd69b640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 0x7f2bc814e6e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2bb802c5d0 tx=0x7f2bc80d2d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8045070 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb8004bf0 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc814ed00 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8001c70 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.038+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc81f7ba0 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.039+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc81f7990 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.039+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2bb8007070 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.040+0000 7f2bbe7fc640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2bac06c9b0 0x7f2bac06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.040+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f2bb80ace30 con 0x7f2bc814e260 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.040+0000 7f2bcde9c640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2bac06c9b0 0x7f2bac06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.041+0000 7f2bcde9c640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2bac06c9b0 0x7f2bac06ee70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2bc80cb000 tx=0x7f2bc4008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.044+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2bc81f88e0 con 0x7f2bc814e260 2024-04-15T15:17:42.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.232+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bc80689e0 con 0x7f2bc814e260 2024-04-15T15:17:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.233+0000 7f2bbe7fc640 1 -- 172.21.15.46:0/523985537 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7f2bb807b180 con 0x7f2bc814e260 2024-04-15T15:17:42.235 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:42.235 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2bac06c9b0 msgr2=0x7f2bac06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2bac06c9b0 0x7f2bac06ee70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2bc80cb000 tx=0x7f2bc4008040 comp rx=0 tx=0).stop 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 msgr2=0x7f2bc814e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 0x7f2bc814e6e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2bb802c5d0 tx=0x7f2bc80d2d20 comp rx=0 tx=0).stop 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 shutdown_connections 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f2bac06c9b0 0x7f2bac06ee70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f2bc80c9ac0 0x7f2bc80c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bc814e260 0x7f2bc814e6e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 --2- 172.21.15.46:0/523985537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb4004800 0x7f2bc81f83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 >> 172.21.15.46:0/523985537 conn(0x7f2bc808bf60 msgr2=0x7f2bc806dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:42.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 shutdown_connections 2024-04-15T15:17:42.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.235+0000 7f2bd0100640 1 -- 172.21.15.46:0/523985537 wait complete. 2024-04-15T15:17:42.248 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:42.248 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:42.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.354+0000 7f33d3274640 1 Processor -- start 2024-04-15T15:17:42.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.354+0000 7f33d3274640 1 -- start start 2024-04-15T15:17:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.355+0000 7f33d3274640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f33cc15e700 0x7f33cc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.355+0000 7f33d3274640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc0b47a0 con 0x7f33cc0d2360 2024-04-15T15:17:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.355+0000 7f33d3274640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33cc0b4a80 con 0x7f33cc0d3270 2024-04-15T15:17:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.355+0000 7f33d3274640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f33cc0b53b0 con 0x7f33cc15e700 2024-04-15T15:17:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.355+0000 7f33d080f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f33cc0d2360 0x7f33cc0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51986/0 (socket says 172.21.15.46:51986) 2024-04-15T15:17:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.355+0000 7f33d080f640 1 -- 172.21.15.46:0/3138943435 learned_addr learned my addr 172.21.15.46:0/3138943435 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224516656 0 0) 0x7f33cc0b47a0 con 0x7f33cc0d2360 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d1010640 1 --2- 172.21.15.46:0/3138943435 >> v2:172.21.15.78:3300/0 conn(0x7f33cc15e700 0x7f33cc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b4003520 con 0x7f33cc0d2360 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2926686299 0 0) 0x7f33b4003520 con 0x7f33cc0d2360 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 >> v2:172.21.15.78:3300/0 conn(0x7f33cc15e700 msgr2=0x7f33cc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 --2- 172.21.15.46:0/3138943435 >> v2:172.21.15.78:3300/0 conn(0x7f33cc15e700 0x7f33cc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 >> v1:172.21.15.78:6789/0 conn(0x7f33cc0d3270 legacy=0x7f33cc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc0b4d60 con 0x7f33cc0d2360 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f33c0001670 con 0x7f33cc0d2360 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 >> v1:172.21.15.46:6789/0 conn(0x7f33cc0d2360 legacy=0x7f33cc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33b40039b0 0x7f33b4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33b40051c0 0x7f33b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0b53b0 con 0x7f33b4004800 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33b400c090 con 0x7f33b40039b0 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f33b400c290 con 0x7f33b40051c0 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d080f640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d1811640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33b40051c0 0x7f33b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d1010640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33b40039b0 0x7f33b4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d080f640 1 -- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33b40051c0 msgr2=0x7f33b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d080f640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33b40051c0 0x7f33b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d080f640 1 -- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33b40039b0 msgr2=0x7f33b4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d080f640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33b40039b0 0x7f33b4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d1811640 1 -- 172.21.15.46:0/3138943435 reap_dead start 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.356+0000 7f33d080f640 1 -- 172.21.15.46:0/3138943435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b4003520 con 0x7f33b4004800 2024-04-15T15:17:42.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.357+0000 7f33d080f640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33b4004c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c240 tx=0x7f33b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80dd21f1e84cb408 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.357+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0007cf0 con 0x7f33b4004800 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.357+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c0030070 con 0x7f33b4004800 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.357+0000 7f33cbfff640 1 -- 172.21.15.46:0/3138943435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0005d30 con 0x7f33b4004800 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 -- 172.21.15.46:0/3138943435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 msgr2=0x7f33b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33b4004c80 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c240 tx=0x7f33b4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 -- 172.21.15.46:0/3138943435 shutdown_connections 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 --2- 172.21.15.46:0/3138943435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33b4004c80 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 -- 172.21.15.46:0/3138943435 >> 172.21.15.46:0/3138943435 conn(0x7f33cc08bf60 msgr2=0x7f33cc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 -- 172.21.15.46:0/3138943435 shutdown_connections 2024-04-15T15:17:42.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.358+0000 7f33d3274640 1 -- 172.21.15.46:0/3138943435 wait complete. 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.359+0000 7f33d3274640 1 Processor -- start 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.359+0000 7f33d3274640 1 -- start start 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33cc14bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33cc14b310 0x7f33cc0cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33cc0ce1c0 0x7f33cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0d1d40 con 0x7f33b4004800 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f33cc0b4450 con 0x7f33cc14b310 2024-04-15T15:17:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f33cc0d1ec0 con 0x7f33cc0ce1c0 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33cc14bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33cc14bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40766/0 (socket says 172.21.15.46:40766) 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 -- 172.21.15.46:0/2718546974 learned_addr learned my addr 172.21.15.46:0/2718546974 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1811640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33cc0ce1c0 0x7f33cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d080f640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33cc14b310 0x7f33cc0cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 -- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33cc0ce1c0 msgr2=0x7f33cc06dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33cc0ce1c0 0x7f33cc06dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 -- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33cc14b310 msgr2=0x7f33cc0cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33cc14b310 0x7f33cc0cdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 -- 172.21.15.46:0/2718546974 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33cc0b2e80 con 0x7f33b4004800 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d080f640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33cc14b310 0x7f33cc0cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1811640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33cc0ce1c0 0x7f33cc06dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d1010640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33cc14bad0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f33c4011500 tx=0x7f33c40119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4019070 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c4005da0 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4006270 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.360+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33cc0b3160 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.361+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33cc0b8ce0 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.362+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33cc0689e0 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.362+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f33c401c070 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.362+0000 7f33c9ffb640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33b806c9b0 0x7f33b806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.363+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f33c408c260 con 0x7f33b4004800 2024-04-15T15:17:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.363+0000 7f33d080f640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33b806c9b0 0x7f33b806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.363+0000 7f33d080f640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33b806c9b0 0x7f33b806ee70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c240 tx=0x7f33c0007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.367+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33c405a420 con 0x7f33b4004800 2024-04-15T15:17:42.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.552+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f33cc0ba500 con 0x7f33b4004800 2024-04-15T15:17:42.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.553+0000 7f33c9ffb640 1 -- 172.21.15.46:0/2718546974 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f33c4059d20 con 0x7f33b4004800 2024-04-15T15:17:42.555 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.554+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33b806c9b0 msgr2=0x7f33b806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.554+0000 7f33d3274640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33b806c9b0 0x7f33b806ee70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f33c000c240 tx=0x7f33c0007000 comp rx=0 tx=0).stop 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.554+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 msgr2=0x7f33cc14bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33cc14bad0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f33c4011500 tx=0x7f33c40119c0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 shutdown_connections 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f33b806c9b0 0x7f33b806ee70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f33cc0ce1c0 0x7f33cc06dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33cc14b310 0x7f33cc0cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 --2- 172.21.15.46:0/2718546974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33b4004800 0x7f33cc14bad0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 >> 172.21.15.46:0/2718546974 conn(0x7f33cc08bf60 msgr2=0x7f33cc0c82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 shutdown_connections 2024-04-15T15:17:42.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.555+0000 7f33d3274640 1 -- 172.21.15.46:0/2718546974 wait complete. 2024-04-15T15:17:42.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.673+0000 7fc6f9a0c640 1 Processor -- start 2024-04-15T15:17:42.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.673+0000 7fc6f9a0c640 1 -- start start 2024-04-15T15:17:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f9a0c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6f4070940 0x7fc6f409b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f9a0c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc6f4160310 0x7fc6f4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f9a0c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f40d05e0 con 0x7fc6f40d0da0 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f9a0c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f4071260 con 0x7fc6f4070940 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f9a0c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f415f720 con 0x7fc6f4160310 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6f4070940 0x7fc6f409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc6f4070940 0x7fc6f409b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40782/0 (socket says 172.21.15.46:40782) 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 -- 172.21.15.46:0/3555790756 learned_addr learned my addr 172.21.15.46:0/3555790756 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6ea7fc640 1 --2- 172.21.15.46:0/3555790756 >> v2:172.21.15.78:3300/0 conn(0x7fc6f4160310 0x7fc6f4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 -- 172.21.15.46:0/3555790756 >> v2:172.21.15.78:3300/0 conn(0x7fc6f4160310 msgr2=0x7fc6f4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 --2- 172.21.15.46:0/3555790756 >> v2:172.21.15.78:3300/0 conn(0x7fc6f4160310 0x7fc6f4070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 -- 172.21.15.46:0/3555790756 >> v1:172.21.15.78:6789/0 conn(0x7fc6f40d0da0 legacy=0x7fc6f40d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.674+0000 7fc6f2ffd640 1 -- 172.21.15.46:0/3555790756 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f415fa00 con 0x7fc6f4070940 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.675+0000 7fc6f2ffd640 1 --2- 172.21.15.46:0/3555790756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4070940 0x7fc6f409b880 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40734f0 tx=0x7fc6e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c53a5121daf97b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.675+0000 7fc6f27fc640 1 -- 172.21.15.46:0/3555790756 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4035020 con 0x7fc6f4070940 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.675+0000 7fc6f27fc640 1 -- 172.21.15.46:0/3555790756 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e4031870 con 0x7fc6f4070940 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.675+0000 7fc6f27fc640 1 -- 172.21.15.46:0/3555790756 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4031c40 con 0x7fc6f4070940 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/3555790756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4070940 msgr2=0x7fc6f409b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/3555790756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4070940 0x7fc6f409b880 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40734f0 tx=0x7fc6e402b910 comp rx=0 tx=0).stop 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/3555790756 shutdown_connections 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/3555790756 >> v2:172.21.15.78:3300/0 conn(0x7fc6f4160310 0x7fc6f4070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/3555790756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4070940 0x7fc6f409b880 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/3555790756 >> 172.21.15.46:0/3555790756 conn(0x7fc6f40b3f40 msgr2=0x7fc6f40b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/3555790756 shutdown_connections 2024-04-15T15:17:42.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.676+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/3555790756 wait complete. 2024-04-15T15:17:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.677+0000 7fc6f9a0c640 1 Processor -- start 2024-04-15T15:17:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.677+0000 7fc6f9a0c640 1 -- start start 2024-04-15T15:17:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f9a0c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6f4070940 0x7fc6f41f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f9a0c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6f40d0da0 0x7fc6f4154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f9a0c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 0x7fc6f40c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f9a0c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f40d18e0 con 0x7fc6f4160310 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f9a0c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f40d0880 con 0x7fc6f4070940 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f9a0c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f4160770 con 0x7fc6f40d0da0 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 0x7fc6f40c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 0x7fc6f40c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40794/0 (socket says 172.21.15.46:40794) 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 -- 172.21.15.46:0/168965200 learned_addr learned my addr 172.21.15.46:0/168965200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6ea7fc640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6f40d0da0 0x7fc6f4154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f2ffd640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6f4070940 0x7fc6f41f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 -- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6f40d0da0 msgr2=0x7fc6f4154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6f40d0da0 0x7fc6f4154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 -- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6f4070940 msgr2=0x7fc6f41f8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6f4070940 0x7fc6f41f8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.678+0000 7fc6f37fe640 1 -- 172.21.15.46:0/168965200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f40b54b0 con 0x7fc6f4160310 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.679+0000 7fc6f37fe640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 0x7fc6f40c65d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00b990 tx=0x7fc6ec00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.679+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec019070 con 0x7fc6f4160310 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.679+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ec00cba0 con 0x7fc6f4160310 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.679+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f40b5790 con 0x7fc6f4160310 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.679+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec014440 con 0x7fc6f4160310 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.679+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f4155970 con 0x7fc6f4160310 2024-04-15T15:17:42.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.680+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f40689e0 con 0x7fc6f4160310 2024-04-15T15:17:42.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.680+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc6ec01c070 con 0x7fc6f4160310 2024-04-15T15:17:42.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.681+0000 7fc6f0ff9640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6d406c980 0x7fc6d406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:42.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.681+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc6ec08baf0 con 0x7fc6f4160310 2024-04-15T15:17:42.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.681+0000 7fc6f2ffd640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6d406c980 0x7fc6d406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:42.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.681+0000 7fc6f2ffd640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6d406c980 0x7fc6d406ee40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4036000 tx=0x7fc6e403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:42.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.685+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6ec059d30 con 0x7fc6f4160310 2024-04-15T15:17:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.874+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6f4070720 con 0x7fc6f4160310 2024-04-15T15:17:42.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.874+0000 7fc6f0ff9640 1 -- 172.21.15.46:0/168965200 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fc6ec059630 con 0x7fc6f4160310 2024-04-15T15:17:42.876 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6d406c980 msgr2=0x7fc6d406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6d406c980 0x7fc6d406ee40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4036000 tx=0x7fc6e403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 msgr2=0x7fc6f40c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 0x7fc6f40c65d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00b990 tx=0x7fc6ec00be50 comp rx=0 tx=0).stop 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 shutdown_connections 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc6d406c980 0x7fc6d406ee40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc6f4160310 0x7fc6f40c65d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc6f40d0da0 0x7fc6f4154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 --2- 172.21.15.46:0/168965200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6f4070940 0x7fc6f41f8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.876+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 >> 172.21.15.46:0/168965200 conn(0x7fc6f40b3f40 msgr2=0x7fc6f409b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.877+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 shutdown_connections 2024-04-15T15:17:42.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.877+0000 7fc6f9a0c640 1 -- 172.21.15.46:0/168965200 wait complete. 2024-04-15T15:17:42.889 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:42.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.995+0000 7f5487bcd640 1 Processor -- start 2024-04-15T15:17:42.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.995+0000 7f5487bcd640 1 -- start start 2024-04-15T15:17:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5487bcd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f54800d3170 0x7f548006d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5487bcd640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54800d0530 con 0x7f54800d15c0 2024-04-15T15:17:43.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5487bcd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54800cf490 con 0x7f548015e7a0 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5487bcd640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f54800cf770 con 0x7f54800d3170 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5485168640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f54800d15c0 0x7f548006cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:51998/0 (socket says 172.21.15.46:51998) 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5485168640 1 -- 172.21.15.46:0/788505646 learned_addr learned my addr 172.21.15.46:0/788505646 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f548616a640 1 --2- 172.21.15.46:0/788505646 >> v2:172.21.15.78:3300/0 conn(0x7f54800d3170 0x7f548006d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252780160 0 0) 0x7f54800d0530 con 0x7f54800d15c0 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.996+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5468003520 con 0x7f54800d15c0 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1089641946 0 0) 0x7f5468003520 con 0x7f54800d15c0 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 >> v2:172.21.15.78:3300/0 conn(0x7f54800d3170 msgr2=0x7f548006d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 --2- 172.21.15.46:0/788505646 >> v2:172.21.15.78:3300/0 conn(0x7f54800d3170 0x7f548006d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 >> v1:172.21.15.78:6789/0 conn(0x7f548015e7a0 legacy=0x7f54800d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54800cfa50 con 0x7f54800d15c0 2024-04-15T15:17:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f5474001670 con 0x7f54800d15c0 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 >> v1:172.21.15.46:6789/0 conn(0x7f54800d15c0 legacy=0x7f548006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54680039b0 0x7f5468003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5468004800 0x7f5468004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680051c0 0x7f546800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54800cf770 con 0x7f54680039b0 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f546800c160 con 0x7f54680051c0 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f546800c360 con 0x7f5468004800 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485969640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54680039b0 0x7f5468003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485168640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5468004800 0x7f5468004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f548616a640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680051c0 0x7f546800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485969640 1 -- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5468004800 msgr2=0x7f5468004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485969640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5468004800 0x7f5468004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485969640 1 -- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680051c0 msgr2=0x7f546800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485969640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680051c0 0x7f546800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485969640 1 -- 172.21.15.46:0/788505646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5468003520 con 0x7f54680039b0 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f548616a640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680051c0 0x7f546800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f5485168640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5468004800 0x7f5468004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.997+0000 7f548616a640 1 -- 172.21.15.46:0/788505646 reap_dead start 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.998+0000 7f5485969640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54680039b0 0x7f5468003dd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5470001e50 tx=0x7f5468005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fa8295613ca74c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.998+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470001a70 con 0x7f54680039b0 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.998+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5470003b70 con 0x7f54680039b0 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.998+0000 7f5484967640 1 -- 172.21.15.46:0/788505646 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54700050d0 con 0x7f54680039b0 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 -- 172.21.15.46:0/788505646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54680039b0 msgr2=0x7f5468003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54680039b0 0x7f5468003dd0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5470001e50 tx=0x7f5468005830 comp rx=0 tx=0).stop 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 -- 172.21.15.46:0/788505646 shutdown_connections 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 --2- 172.21.15.46:0/788505646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54680039b0 0x7f5468003dd0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 -- 172.21.15.46:0/788505646 >> 172.21.15.46:0/788505646 conn(0x7f548008bf60 msgr2=0x7f54800d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 -- 172.21.15.46:0/788505646 shutdown_connections 2024-04-15T15:17:43.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:42.999+0000 7f5487bcd640 1 -- 172.21.15.46:0/788505646 wait complete. 2024-04-15T15:17:43.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.000+0000 7f5487bcd640 1 Processor -- start 2024-04-15T15:17:43.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.000+0000 7f5487bcd640 1 -- start start 2024-04-15T15:17:43.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680039b0 0x7f54800c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54800cbce0 0x7f54800cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 0x7f54800ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f548015eb80 con 0x7f54800ce250 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f54800cf180 con 0x7f54680039b0 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f54800d1a40 con 0x7f54800cbce0 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 0x7f54800ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 0x7f54800ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40822/0 (socket says 172.21.15.46:40822) 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 -- 172.21.15.46:0/2929925719 learned_addr learned my addr 172.21.15.46:0/2929925719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5485168640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54800cbce0 0x7f54800cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5485969640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680039b0 0x7f54800c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 -- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54800cbce0 msgr2=0x7f54800cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54800cbce0 0x7f54800cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 -- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680039b0 msgr2=0x7f54800c9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680039b0 0x7f54800c9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 -- 172.21.15.46:0/2929925719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54801f4470 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f548616a640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 0x7f54800ce6d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f54800cef70 tx=0x7f5478011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478003610 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f547801b070 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478003a10 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54800cc8c0 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.001+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54801f4a10 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.002+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54800689e0 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.003+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5478003bb0 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.003+0000 7f547e7fc640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f546c06c980 0x7f546c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.004+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f547808c2a0 con 0x7f54800ce250 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.004+0000 7f5485969640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f546c06c980 0x7f546c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.004+0000 7f5485969640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f546c06c980 0x7f546c06ee40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5470034000 tx=0x7f5470009030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.008+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f547805a3e0 con 0x7f54800ce250 2024-04-15T15:17:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.192+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f54800d1d60 con 0x7f54800ce250 2024-04-15T15:17:43.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.192+0000 7f547e7fc640 1 -- 172.21.15.46:0/2929925719 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f5478059ce0 con 0x7f54800ce250 2024-04-15T15:17:43.195 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f546c06c980 msgr2=0x7f546c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f546c06c980 0x7f546c06ee40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5470034000 tx=0x7f5470009030 comp rx=0 tx=0).stop 2024-04-15T15:17:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 msgr2=0x7f54800ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 0x7f54800ce6d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f54800cef70 tx=0x7f5478011920 comp rx=0 tx=0).stop 2024-04-15T15:17:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 shutdown_connections 2024-04-15T15:17:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f546c06c980 0x7f546c06ee40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54800ce250 0x7f54800ce6d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f54800cbce0 0x7f54800cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 --2- 172.21.15.46:0/2929925719 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54680039b0 0x7f54800c9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 >> 172.21.15.46:0/2929925719 conn(0x7f548008bf60 msgr2=0x7f54800b19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 shutdown_connections 2024-04-15T15:17:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.194+0000 7f5487bcd640 1 -- 172.21.15.46:0/2929925719 wait complete. 2024-04-15T15:17:43.207 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:43.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.311+0000 7ff605304640 1 Processor -- start 2024-04-15T15:17:43.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.311+0000 7ff605304640 1 -- start start 2024-04-15T15:17:43.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff605304640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff600070940 0x7ff60009b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff605304640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 0x7ff600070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff605304640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6000d05e0 con 0x7ff6000d0da0 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff605304640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7ff600071260 con 0x7ff600070940 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff605304640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff6000d1940 con 0x7ff600160310 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5ff7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff6000d0da0 0x7ff6000d1230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52014/0 (socket says 172.21.15.46:52014) 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5ff7fe640 1 -- 172.21.15.46:0/4262690323 learned_addr learned my addr 172.21.15.46:0/4262690323 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5fe7fc640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 0x7ff600070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897769358 0 0) 0x7ff6000d05e0 con 0x7ff6000d0da0 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5feffd640 1 --2- 172.21.15.46:0/4262690323 >> v2:172.21.15.78:3300/0 conn(0x7ff600070940 0x7ff60009b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5ec003520 con 0x7ff6000d0da0 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2840738332 0 0) 0x7ff5ec003520 con 0x7ff6000d0da0 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 msgr2=0x7ff600070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.312+0000 7ff5fdffb640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 0x7ff600070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 >> v2:172.21.15.78:3300/0 conn(0x7ff600070940 msgr2=0x7ff60009b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 --2- 172.21.15.46:0/4262690323 >> v2:172.21.15.78:3300/0 conn(0x7ff600070940 0x7ff60009b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60015f720 con 0x7ff6000d0da0 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7ff5f8001670 con 0x7ff6000d0da0 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 >> v1:172.21.15.46:6789/0 conn(0x7ff6000d0da0 legacy=0x7ff6000d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ec0039b0 0x7ff5ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec004800 0x7ff5ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 0x7ff5ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff600071260 con 0x7ff5ec0039b0 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff6000d1940 con 0x7ff5ec004800 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff5ec00b9c0 con 0x7ff600160310 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5feffd640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ec0039b0 0x7ff5ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5fe7fc640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec004800 0x7ff5ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5ff7fe640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 0x7ff5ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5feffd640 1 -- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 msgr2=0x7ff5ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5feffd640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff600160310 0x7ff5ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5feffd640 1 -- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec004800 msgr2=0x7ff5ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5feffd640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec004800 0x7ff5ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.313+0000 7ff5feffd640 1 -- 172.21.15.46:0/4262690323 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec003520 con 0x7ff5ec0039b0 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff5feffd640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ec0039b0 0x7ff5ec003dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009fc0 tx=0x7ff5ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7891bb0b6fe44b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0039070 con 0x7ff5ec0039b0 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f002cc60 con 0x7ff5ec0039b0 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff5fdffb640 1 -- 172.21.15.46:0/4262690323 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0009c10 con 0x7ff5ec0039b0 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff605304640 1 -- 172.21.15.46:0/4262690323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ec0039b0 msgr2=0x7ff5ec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff605304640 1 --2- 172.21.15.46:0/4262690323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5ec0039b0 0x7ff5ec003dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009fc0 tx=0x7ff5ec005830 comp rx=0 tx=0).stop 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.314+0000 7ff5ff7fe640 1 -- 172.21.15.46:0/4262690323 reap_dead start 2024-04-15T15:17:43.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.315+0000 7ff605304640 1 -- 172.21.15.46:0/4262690323 shutdown_connections 2024-04-15T15:17:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.315+0000 7ff605304640 1 -- 172.21.15.46:0/4262690323 >> 172.21.15.46:0/4262690323 conn(0x7ff6000b3f40 msgr2=0x7ff6000b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.315+0000 7ff605304640 1 -- 172.21.15.46:0/4262690323 shutdown_connections 2024-04-15T15:17:43.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.315+0000 7ff605304640 1 -- 172.21.15.46:0/4262690323 wait complete. 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.316+0000 7ff605304640 1 Processor -- start 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.316+0000 7ff605304640 1 -- start start 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec0039b0 0x7ff600153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6000b52a0 0x7ff6000b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 0x7ff6000b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff6000d0880 con 0x7ff6000b7460 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff60015fce0 con 0x7ff5ec0039b0 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff600160770 con 0x7ff6000b52a0 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 0x7ff6000b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5feffd640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec0039b0 0x7ff600153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 0x7ff6000b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40838/0 (socket says 172.21.15.46:40838) 2024-04-15T15:17:43.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 -- 172.21.15.46:0/3471206997 learned_addr learned my addr 172.21.15.46:0/3471206997 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5fe7fc640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6000b52a0 0x7ff6000b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 -- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6000b52a0 msgr2=0x7ff6000b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6000b52a0 0x7ff6000b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 -- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec0039b0 msgr2=0x7ff600153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec0039b0 0x7ff600153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 -- 172.21.15.46:0/3471206997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6001580c0 con 0x7ff6000b7460 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5feffd640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec0039b0 0x7ff600153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5ff7fe640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 0x7ff6000b78e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8001050 tx=0x7ff5f8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f801a070 con 0x7ff6000b7460 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f800f730 con 0x7ff6000b7460 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.317+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6000b5df0 con 0x7ff6000b7460 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.318+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800fb00 con 0x7ff6000b7460 2024-04-15T15:17:43.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.318+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6000b63b0 con 0x7ff6000b7460 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.319+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6000689e0 con 0x7ff6000b7460 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.319+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff5f801d070 con 0x7ff6000b7460 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.319+0000 7ff5f77fe640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff5e006c980 0x7ff5e006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.320+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7ff5f808b190 con 0x7ff6000b7460 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.320+0000 7ff5feffd640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff5e006c980 0x7ff5e006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.320+0000 7ff5feffd640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff5e006c980 0x7ff5e006ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0036000 tx=0x7ff5f002e000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.324+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff5f8058d80 con 0x7ff6000b7460 2024-04-15T15:17:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.521+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff600070720 con 0x7ff6000b7460 2024-04-15T15:17:43.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.522+0000 7ff5f77fe640 1 -- 172.21.15.46:0/3471206997 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+120 (secure 0 0 0) 0x7ff5f805d080 con 0x7ff6000b7460 2024-04-15T15:17:43.524 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6835/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:43.524 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6833/964535021 2024-04-16T15:17:30.625393+0000 2024-04-15T15:17:43.524 INFO:teuthology.orchestra.run.smithi046.stderr:listed 2 entries 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff5e006c980 msgr2=0x7ff5e006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff5e006c980 0x7ff5e006ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0036000 tx=0x7ff5f002e000 comp rx=0 tx=0).stop 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 msgr2=0x7ff6000b78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 0x7ff6000b78e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8001050 tx=0x7ff5f8007430 comp rx=0 tx=0).stop 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 shutdown_connections 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff5e006c980 0x7ff5e006ee40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6000b7460 0x7ff6000b78e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff6000b52a0 0x7ff6000b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 --2- 172.21.15.46:0/3471206997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5ec0039b0 0x7ff600153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 >> 172.21.15.46:0/3471206997 conn(0x7ff6000b3f40 msgr2=0x7ff6000b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 shutdown_connections 2024-04-15T15:17:43.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.524+0000 7ff605304640 1 -- 172.21.15.46:0/3471206997 wait complete. 2024-04-15T15:17:43.537 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2024-04-15T15:17:43.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:43.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:17:43.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi078.stderr:ceph-fuse[45585]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:43.571 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:17:43.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:43.571 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:17:43.744 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.849+0000 7f40ec3aa640 1 Processor -- start 2024-04-15T15:17:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.849+0000 7f40ec3aa640 1 -- start start 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.850+0000 7f40ec3aa640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f40e409b880 0x7f40e4070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.850+0000 7f40ec3aa640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e40d05e0 con 0x7f40e4070940 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.850+0000 7f40ec3aa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40e4071260 con 0x7f40e40d0da0 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.850+0000 7f40ec3aa640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f40e409bce0 con 0x7f40e409b880 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.850+0000 7f40ea146640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f40e4070940 0x7f40e4160310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52028/0 (socket says 172.21.15.46:52028) 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.850+0000 7f40ea146640 1 -- 172.21.15.46:0/3020006791 learned_addr learned my addr 172.21.15.46:0/3020006791 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:43.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698996461 0 0) 0x7f40e40d05e0 con 0x7f40e4070940 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40cc003520 con 0x7f40e4070940 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 --2- 172.21.15.46:0/3020006791 >> v2:172.21.15.78:3300/0 conn(0x7f40e409b880 0x7f40e4070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2512305138 0 0) 0x7f40cc003520 con 0x7f40e4070940 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 >> v2:172.21.15.78:3300/0 conn(0x7f40e409b880 msgr2=0x7f40e4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 --2- 172.21.15.46:0/3020006791 >> v2:172.21.15.78:3300/0 conn(0x7f40e409b880 0x7f40e4070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 >> v1:172.21.15.78:6789/0 conn(0x7f40e40d0da0 legacy=0x7f40e40d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e415f720 con 0x7f40e4070940 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f40dc001670 con 0x7f40e4070940 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 >> v1:172.21.15.46:6789/0 conn(0x7f40e4070940 legacy=0x7f40e4160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40cc0039b0 0x7f40cc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc004800 0x7f40cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc0051c0 0x7f40cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40e409bce0 con 0x7f40cc004800 2024-04-15T15:17:43.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc00c0f0 con 0x7f40cc0039b0 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc00c2f0 con 0x7f40cc0051c0 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc004800 0x7f40cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40ea146640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40cc0039b0 0x7f40cc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40ea947640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc0051c0 0x7f40cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 -- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc0051c0 msgr2=0x7f40cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc0051c0 0x7f40cc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 -- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40cc0039b0 msgr2=0x7f40cc003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40cc0039b0 0x7f40cc003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40e9945640 1 -- 172.21.15.46:0/3020006791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc003520 con 0x7f40cc004800 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.851+0000 7f40ea947640 1 -- 172.21.15.46:0/3020006791 reap_dead start 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.852+0000 7f40e9945640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc004800 0x7f40cc004c80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f40d8007330 tx=0x7f40cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29a4b0f3505b770 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.852+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8039070 con 0x7f40cc004800 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.852+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d802cc70 con 0x7f40cc004800 2024-04-15T15:17:43.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.852+0000 7f40e9144640 1 -- 172.21.15.46:0/3020006791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8009be0 con 0x7f40cc004800 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 -- 172.21.15.46:0/3020006791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc004800 msgr2=0x7f40cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc004800 0x7f40cc004c80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f40d8007330 tx=0x7f40cc005830 comp rx=0 tx=0).stop 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 -- 172.21.15.46:0/3020006791 shutdown_connections 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/3020006791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40cc004800 0x7f40cc004c80 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 -- 172.21.15.46:0/3020006791 >> 172.21.15.46:0/3020006791 conn(0x7f40e40b3f40 msgr2=0x7f40e40b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 -- 172.21.15.46:0/3020006791 shutdown_connections 2024-04-15T15:17:43.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.853+0000 7f40ec3aa640 1 -- 172.21.15.46:0/3020006791 wait complete. 2024-04-15T15:17:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.854+0000 7f40ec3aa640 1 Processor -- start 2024-04-15T15:17:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.854+0000 7f40ec3aa640 1 -- start start 2024-04-15T15:17:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ec3aa640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc004800 0x7f40e40b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ec3aa640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 0x7f40e40b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ec3aa640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40b7810 0x7f40e4158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ec3aa640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f40e40d0880 con 0x7f40e4157cf0 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ec3aa640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f40e415fce0 con 0x7f40e40b7810 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ec3aa640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f40e4070d20 con 0x7f40cc004800 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 0x7f40e40b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 0x7f40e40b72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40854/0 (socket says 172.21.15.46:40854) 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 -- 172.21.15.46:0/2170383434 learned_addr learned my addr 172.21.15.46:0/2170383434 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:43.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ea146640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc004800 0x7f40e40b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ea947640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40b7810 0x7f40e4158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 -- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc004800 msgr2=0x7f40e40b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc004800 0x7f40e40b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 -- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40b7810 msgr2=0x7f40e4158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40b7810 0x7f40e4158540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 -- 172.21.15.46:0/2170383434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40e40b57e0 con 0x7f40e4157cf0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40ea947640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40b7810 0x7f40e4158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.855+0000 7f40e9945640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 0x7f40e40b72d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f40d8000f80 tx=0x7f40d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.856+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8042030 con 0x7f40e4157cf0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.856+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d803a040 con 0x7f40e4157cf0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.856+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8039070 con 0x7f40e4157cf0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.856+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40e4159120 con 0x7f40e4157cf0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.856+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40e41596e0 con 0x7f40e4157cf0 2024-04-15T15:17:43.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.856+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40e40d0ac0 con 0x7f40e4157cf0 2024-04-15T15:17:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.857+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f40d8045070 con 0x7f40e4157cf0 2024-04-15T15:17:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.857+0000 7f40e2ffd640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f40d006c9b0 0x7f40d006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.858+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f40d80ad980 con 0x7f40e4157cf0 2024-04-15T15:17:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.858+0000 7f40ea146640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f40d006c9b0 0x7f40d006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.858+0000 7f40ea146640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f40d006c9b0 0x7f40d006ee70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f40e406a7b0 tx=0x7f40dc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:43.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:43.862+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40e40d0ac0 con 0x7f40e4157cf0 2024-04-15T15:17:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.083+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40e409b0b0 con 0x7f40e4157cf0 2024-04-15T15:17:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.084+0000 7f40e2ffd640 1 -- 172.21.15.46:0/2170383434 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5176 (secure 0 0 0) 0x7f40e409b0b0 con 0x7f40e4157cf0 2024-04-15T15:17:44.086 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:44.086 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:34.636332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5946,"qdb_cluster":[5946]},"id":2}]} 2024-04-15T15:17:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-15T15:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f40d006c9b0 msgr2=0x7f40d006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f40d006c9b0 0x7f40d006ee70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f40e406a7b0 tx=0x7f40dc009040 comp rx=0 tx=0).stop 2024-04-15T15:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 msgr2=0x7f40e40b72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 0x7f40e40b72d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f40d8000f80 tx=0x7f40d8004ab0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 shutdown_connections 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f40d006c9b0 0x7f40d006ee70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f40e40b7810 0x7f40e4158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40e4157cf0 0x7f40e40b72d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.085+0000 7f40ec3aa640 1 --2- 172.21.15.46:0/2170383434 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f40cc004800 0x7f40e40b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.086+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 >> 172.21.15.46:0/2170383434 conn(0x7f40e40b3f40 msgr2=0x7f40e406e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.086+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 shutdown_connections 2024-04-15T15:17:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.086+0000 7f40ec3aa640 1 -- 172.21.15.46:0/2170383434 wait complete. 2024-04-15T15:17:44.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:44.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.202+0000 7f7a82c77640 1 Processor -- start 2024-04-15T15:17:44.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.202+0000 7f7a82c77640 1 -- start start 2024-04-15T15:17:44.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a82c77640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a7c0d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a82c77640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a7c15eb70 con 0x7f7a7c15e790 2024-04-15T15:17:44.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a82c77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a7c0d3c80 con 0x7f7a7c0d15e0 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a82c77640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7c0d3790 con 0x7f7a7c0d5150 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a80a13640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7a7c15e790 0x7f7a7c0b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52032/0 (socket says 172.21.15.46:52032) 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a80a13640 1 -- 172.21.15.46:0/2151870560 learned_addr learned my addr 172.21.15.46:0/2151870560 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a81214640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a7c0d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825620918 0 0) 0x7f7a7c15eb70 con 0x7f7a7c15e790 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a64003520 con 0x7f7a7c15e790 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1657938261 0 0) 0x7f7a7c0d3c80 con 0x7f7a7c0d15e0 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a7c15eb70 con 0x7f7a7c0d15e0 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 169766449 0 0) 0x7f7a64003520 con 0x7f7a7c15e790 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 msgr2=0x7f7a7c0d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a7c0d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 >> v1:172.21.15.78:6789/0 conn(0x7f7a7c0d15e0 legacy=0x7f7a7c0d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a7c0d3f60 con 0x7f7a7c15e790 2024-04-15T15:17:44.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f7a78001670 con 0x7f7a7c15e790 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.203+0000 7f7a81214640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a7c0d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 >> v1:172.21.15.46:6789/0 conn(0x7f7a7c15e790 legacy=0x7f7a7c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a64003b70 0x7f7a64003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a64004d40 0x7f7a640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7c0d3790 con 0x7f7a64003b70 2024-04-15T15:17:44.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6400bb50 con 0x7f7a64004d40 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7a6400bd50 con 0x7f7a7c0d5150 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a80a13640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a64003b70 0x7f7a64003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a73fff640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a81214640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a64004d40 0x7f7a640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a80a13640 1 -- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 msgr2=0x7f7a64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a80a13640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a7c0d5150 0x7f7a64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a80a13640 1 -- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a64004d40 msgr2=0x7f7a640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a80a13640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a64004d40 0x7f7a640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.204+0000 7f7a80a13640 1 -- 172.21.15.46:0/2151870560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a64003520 con 0x7f7a64003b70 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a80a13640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a64003b70 0x7f7a64003f70 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800c190 tx=0x7f7a64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe1a8815076b4f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78007cc0 con 0x7f7a64003b70 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a78030070 con 0x7f7a64003b70 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a737fe640 1 -- 172.21.15.46:0/2151870560 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78005d30 con 0x7f7a64003b70 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a82c77640 1 -- 172.21.15.46:0/2151870560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a64003b70 msgr2=0x7f7a64003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a82c77640 1 --2- 172.21.15.46:0/2151870560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a64003b70 0x7f7a64003f70 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800c190 tx=0x7f7a64005830 comp rx=0 tx=0).stop 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a81214640 1 -- 172.21.15.46:0/2151870560 reap_dead start 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a82c77640 1 -- 172.21.15.46:0/2151870560 shutdown_connections 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a82c77640 1 -- 172.21.15.46:0/2151870560 >> 172.21.15.46:0/2151870560 conn(0x7f7a7c067b40 msgr2=0x7f7a7c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a82c77640 1 -- 172.21.15.46:0/2151870560 shutdown_connections 2024-04-15T15:17:44.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.205+0000 7f7a82c77640 1 -- 172.21.15.46:0/2151870560 wait complete. 2024-04-15T15:17:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 Processor -- start 2024-04-15T15:17:44.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 -- start start 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a64003b70 0x7f7a7c0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0ab920 0x7f7a7c1504b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 0x7f7a7c0ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7c0b7db0 con 0x7f7a7c0ae310 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7c0d45b0 con 0x7f7a7c0ab920 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.207+0000 7f7a82c77640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7c0d1a90 con 0x7f7a64003b70 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 0x7f7a7c0ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 0x7f7a7c0ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40874/0 (socket says 172.21.15.46:40874) 2024-04-15T15:17:44.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 -- 172.21.15.46:0/1023762907 learned_addr learned my addr 172.21.15.46:0/1023762907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a80a13640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a64003b70 0x7f7a7c0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a73fff640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0ab920 0x7f7a7c1504b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 -- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a64003b70 msgr2=0x7f7a7c0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a64003b70 0x7f7a7c0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 -- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0ab920 msgr2=0x7f7a7c1504b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0ab920 0x7f7a7c1504b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 -- 172.21.15.46:0/1023762907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a7c0ac670 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a81214640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 0x7f7a7c0ae790 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0d2440 tx=0x7f7a74009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400fd70 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a74003990 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a74003d80 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a7c0ac950 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.208+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a7c0acf10 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.209+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7c0689e0 con 0x7f7a7c0ae310 2024-04-15T15:17:44.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.210+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7a74004020 con 0x7f7a7c0ae310 2024-04-15T15:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.210+0000 7f7a717fa640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7a6806c9b0 0x7f7a6806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.210+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f7a7408caa0 con 0x7f7a7c0ae310 2024-04-15T15:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.210+0000 7f7a80a13640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7a6806c9b0 0x7f7a6806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.211+0000 7f7a80a13640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7a6806c9b0 0x7f7a6806ee70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7a78034000 tx=0x7f7a78007000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.215+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a74019030 con 0x7f7a7c0ae310 2024-04-15T15:17:44.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.399+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a7c0d0100 con 0x7f7a7c0ae310 2024-04-15T15:17:44.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.400+0000 7f7a717fa640 1 -- 172.21.15.46:0/1023762907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7f7a74010040 con 0x7f7a7c0ae310 2024-04-15T15:17:44.402 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:44.402 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7a6806c9b0 msgr2=0x7f7a6806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7a6806c9b0 0x7f7a6806ee70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7a78034000 tx=0x7f7a78007000 comp rx=0 tx=0).stop 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 msgr2=0x7f7a7c0ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 0x7f7a7c0ae790 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c0d2440 tx=0x7f7a74009f90 comp rx=0 tx=0).stop 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 shutdown_connections 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7a6806c9b0 0x7f7a6806ee70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a7c0ae310 0x7f7a7c0ae790 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a7c0ab920 0x7f7a7c1504b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 --2- 172.21.15.46:0/1023762907 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7a64003b70 0x7f7a7c0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 >> 172.21.15.46:0/1023762907 conn(0x7f7a7c067b40 msgr2=0x7f7a7c0cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:44.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 shutdown_connections 2024-04-15T15:17:44.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.402+0000 7f7a82c77640 1 -- 172.21.15.46:0/1023762907 wait complete. 2024-04-15T15:17:44.415 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:15:26.367193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:58.861526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:59.361892+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:44.415 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:44.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.522+0000 7f5cf6952640 1 Processor -- start 2024-04-15T15:17:44.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.522+0000 7f5cf6952640 1 -- start start 2024-04-15T15:17:44.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.522+0000 7f5cf6952640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015e700 0x7f5cf00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.522+0000 7f5cf6952640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f5cf00d2360 0x7f5cf00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf6952640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cf015eae0 0x7f5cf00d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf6952640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf00b53b0 con 0x7f5cf015eae0 2024-04-15T15:17:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf6952640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf00b47a0 con 0x7f5cf00d2360 2024-04-15T15:17:44.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf6952640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00b4a80 con 0x7f5cf015e700 2024-04-15T15:17:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cf015eae0 0x7f5cf00d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5cf015eae0 0x7f5cf00d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40876/0 (socket says 172.21.15.46:40876) 2024-04-15T15:17:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 -- 172.21.15.46:0/2279816506 learned_addr learned my addr 172.21.15.46:0/2279816506 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5ceffff640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015e700 0x7f5cf00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5ce7fff640 1 --2- 172.21.15.46:0/2279816506 >> v2:172.21.15.78:3300/0 conn(0x7f5cf00d2360 0x7f5cf00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 -- 172.21.15.46:0/2279816506 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015e700 msgr2=0x7f5cf00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015e700 0x7f5cf00b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 -- 172.21.15.46:0/2279816506 >> v2:172.21.15.78:3300/0 conn(0x7f5cf00d2360 msgr2=0x7f5cf00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:44.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 --2- 172.21.15.46:0/2279816506 >> v2:172.21.15.78:3300/0 conn(0x7f5cf00d2360 0x7f5cf00d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5cf4eef640 1 -- 172.21.15.46:0/2279816506 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00b4d60 con 0x7f5cf015eae0 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5ceffff640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015e700 0x7f5cf00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.523+0000 7f5ce7fff640 1 --2- 172.21.15.46:0/2279816506 >> v2:172.21.15.78:3300/0 conn(0x7f5cf00d2360 0x7f5cf00d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.524+0000 7f5cf4eef640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf015eae0 0x7f5cf00d30a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00d2100 tx=0x7f5ce802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c51a684d7d18654 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.524+0000 7f5cef7fe640 1 -- 172.21.15.46:0/2279816506 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce802ca70 con 0x7f5cf015eae0 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.524+0000 7f5cef7fe640 1 -- 172.21.15.46:0/2279816506 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce802cc10 con 0x7f5cf015eae0 2024-04-15T15:17:44.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.524+0000 7f5cef7fe640 1 -- 172.21.15.46:0/2279816506 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80343f0 con 0x7f5cf015eae0 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 -- 172.21.15.46:0/2279816506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf015eae0 msgr2=0x7f5cf00d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf015eae0 0x7f5cf00d30a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00d2100 tx=0x7f5ce802bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 -- 172.21.15.46:0/2279816506 shutdown_connections 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf015eae0 0x7f5cf00d30a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 --2- 172.21.15.46:0/2279816506 >> v2:172.21.15.78:3300/0 conn(0x7f5cf00d2360 0x7f5cf00d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 --2- 172.21.15.46:0/2279816506 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015e700 0x7f5cf00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 -- 172.21.15.46:0/2279816506 >> 172.21.15.46:0/2279816506 conn(0x7f5cf008bf60 msgr2=0x7f5cf00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 -- 172.21.15.46:0/2279816506 shutdown_connections 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.525+0000 7f5cf6952640 1 -- 172.21.15.46:0/2279816506 wait complete. 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.526+0000 7f5cf6952640 1 Processor -- start 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.526+0000 7f5cf6952640 1 -- start start 2024-04-15T15:17:44.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 0x7f5cf0160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf015e700 0x7f5cf00c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015eae0 0x7f5cf00c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00b5d80 con 0x7f5cf00d2360 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00b4450 con 0x7f5cf015e700 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00d1d40 con 0x7f5cf015eae0 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 0x7f5cf0160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 0x7f5cf0160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40888/0 (socket says 172.21.15.46:40888) 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 -- 172.21.15.46:0/60243299 learned_addr learned my addr 172.21.15.46:0/60243299 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf4eef640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015eae0 0x7f5cf00c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ce7fff640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf015e700 0x7f5cf00c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 -- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015eae0 msgr2=0x7f5cf00c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015eae0 0x7f5cf00c8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 -- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf015e700 msgr2=0x7f5cf00c7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf015e700 0x7f5cf00c7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 -- 172.21.15.46:0/60243299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf0160920 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ceffff640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 0x7f5cf0160020 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c7f0 tx=0x7f5ce000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0005b90 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce0005d30 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf0160c00 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0007a30 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.527+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf01611c0 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.528+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf00689e0 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.529+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5ce0007bd0 con 0x7f5cf00d2360 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.529+0000 7f5ced7fa640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cd006c9b0 0x7f5cd006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.530+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f5ce008bf30 con 0x7f5cf00d2360 2024-04-15T15:17:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.530+0000 7f5ce7fff640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cd006c9b0 0x7f5cd006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.530+0000 7f5ce7fff640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cd006c9b0 0x7f5cd006ee70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00a23f0 tx=0x7f5cd8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.534+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ce0015020 con 0x7f5cf00d2360 2024-04-15T15:17:44.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.719+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5cf00ba500 con 0x7f5cf00d2360 2024-04-15T15:17:44.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.719+0000 7f5ced7fa640 1 -- 172.21.15.46:0/60243299 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f5ce005a060 con 0x7f5cf00d2360 2024-04-15T15:17:44.721 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cd006c9b0 msgr2=0x7f5cd006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cd006c9b0 0x7f5cd006ee70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00a23f0 tx=0x7f5cd8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 msgr2=0x7f5cf0160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 0x7f5cf0160020 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c7f0 tx=0x7f5ce000cbf0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 shutdown_connections 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5cd006c9b0 0x7f5cd006ee70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5cf015eae0 0x7f5cf00c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5cf015e700 0x7f5cf00c7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 --2- 172.21.15.46:0/60243299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5cf00d2360 0x7f5cf0160020 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 >> 172.21.15.46:0/60243299 conn(0x7f5cf008bf60 msgr2=0x7f5cf00d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:44.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 shutdown_connections 2024-04-15T15:17:44.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.721+0000 7f5cf6952640 1 -- 172.21.15.46:0/60243299 wait complete. 2024-04-15T15:17:44.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:44.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 Processor -- start 2024-04-15T15:17:44.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 -- start start 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f854015ea40 0x7f85400b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f85400d3480 0x7f85400b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85400cff70 con 0x7f854015ee20 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f85400d0250 con 0x7f854015ea40 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.841+0000 7f8546359640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f85400d0530 con 0x7f85400d3480 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f854015ea40 0x7f85400b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f854015ee20 0x7f85400b6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52042/0 (socket says 172.21.15.46:52042) 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853f7fe640 1 -- 172.21.15.46:0/3923418094 learned_addr learned my addr 172.21.15.46:0/3923418094 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:44.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f85448f6640 1 --2- 172.21.15.46:0/3923418094 >> v2:172.21.15.78:3300/0 conn(0x7f85400d3480 0x7f85400b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853ffff640 1 -- 172.21.15.46:0/3923418094 >> v2:172.21.15.78:3300/0 conn(0x7f85400d3480 msgr2=0x7f85400b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:44.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853ffff640 1 --2- 172.21.15.46:0/3923418094 >> v2:172.21.15.78:3300/0 conn(0x7f85400d3480 0x7f85400b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853ffff640 1 -- 172.21.15.46:0/3923418094 >> v1:172.21.15.46:6789/0 conn(0x7f854015ee20 legacy=0x7f85400b6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.842+0000 7f853ffff640 1 -- 172.21.15.46:0/3923418094 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85400d0810 con 0x7f854015ea40 2024-04-15T15:17:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f853ffff640 1 --2- 172.21.15.46:0/3923418094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f85400b5b40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f85400d0f90 tx=0x7f853002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7a1fbf8f8cdbc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f853effd640 1 -- 172.21.15.46:0/3923418094 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530038070 con 0x7f854015ea40 2024-04-15T15:17:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f853effd640 1 -- 172.21.15.46:0/3923418094 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8530033640 con 0x7f854015ea40 2024-04-15T15:17:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f853effd640 1 -- 172.21.15.46:0/3923418094 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8530033a10 con 0x7f854015ea40 2024-04-15T15:17:44.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 -- 172.21.15.46:0/3923418094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 msgr2=0x7f85400b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 --2- 172.21.15.46:0/3923418094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f85400b5b40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f85400d0f90 tx=0x7f853002b7a0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 -- 172.21.15.46:0/3923418094 shutdown_connections 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 --2- 172.21.15.46:0/3923418094 >> v2:172.21.15.78:3300/0 conn(0x7f85400d3480 0x7f85400b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 --2- 172.21.15.46:0/3923418094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f85400b5b40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 -- 172.21.15.46:0/3923418094 >> 172.21.15.46:0/3923418094 conn(0x7f854008bf60 msgr2=0x7f85400d4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.843+0000 7f8546359640 1 -- 172.21.15.46:0/3923418094 shutdown_connections 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.844+0000 7f8546359640 1 -- 172.21.15.46:0/3923418094 wait complete. 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.845+0000 7f8546359640 1 Processor -- start 2024-04-15T15:17:44.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.845+0000 7f8546359640 1 -- start start 2024-04-15T15:17:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85400d3480 0x7f85400cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f854015c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f854015ee20 0x7f8540153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f85400b3a40 con 0x7f854015ea40 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f85400b38c0 con 0x7f854015ee20 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f85400b6790 con 0x7f85400d3480 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f854015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f854015c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40894/0 (socket says 172.21.15.46:40894) 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 -- 172.21.15.46:0/3770717564 learned_addr learned my addr 172.21.15.46:0/3770717564 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f85448f6640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f854015ee20 0x7f8540153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853ffff640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85400d3480 0x7f85400cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 -- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85400d3480 msgr2=0x7f85400cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85400d3480 0x7f85400cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 -- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f854015ee20 msgr2=0x7f8540153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f854015ee20 0x7f8540153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:44.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 -- 172.21.15.46:0/3770717564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85400cd590 con 0x7f854015ea40 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853f7fe640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f854015c360 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f853400c780 tx=0x7f853400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534018070 con 0x7f854015ea40 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85340072d0 con 0x7f854015ea40 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85400b8ae0 con 0x7f854015ea40 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85340047e0 con 0x7f854015ea40 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.846+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85400b90a0 con 0x7f854015ea40 2024-04-15T15:17:44.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.847+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85400b9870 con 0x7f854015ea40 2024-04-15T15:17:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.848+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f853401b070 con 0x7f854015ea40 2024-04-15T15:17:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.848+0000 7f853cff9640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f852c06c980 0x7f852c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.849+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f853408b1b0 con 0x7f854015ea40 2024-04-15T15:17:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.849+0000 7f853ffff640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f852c06c980 0x7f852c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:44.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.849+0000 7f853ffff640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f852c06c980 0x7f852c06ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8530035000 tx=0x7f853002f000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:44.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:44.853+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8534059480 con 0x7f854015ea40 2024-04-15T15:17:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.035+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f85400689e0 con 0x7f854015ea40 2024-04-15T15:17:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.036+0000 7f853cff9640 1 -- 172.21.15.46:0/3770717564 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f8534058d80 con 0x7f854015ea40 2024-04-15T15:17:45.038 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f852c06c980 msgr2=0x7f852c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f852c06c980 0x7f852c06ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8530035000 tx=0x7f853002f000 comp rx=0 tx=0).stop 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 msgr2=0x7f854015c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f854015c360 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f853400c780 tx=0x7f853400cc40 comp rx=0 tx=0).stop 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 shutdown_connections 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f852c06c980 0x7f852c06ee40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f854015ee20 0x7f8540153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f854015ea40 0x7f854015c360 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 --2- 172.21.15.46:0/3770717564 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f85400d3480 0x7f85400cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 >> 172.21.15.46:0/3770717564 conn(0x7f854008bf60 msgr2=0x7f85400b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 shutdown_connections 2024-04-15T15:17:45.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.038+0000 7f8546359640 1 -- 172.21.15.46:0/3770717564 wait complete. 2024-04-15T15:17:45.051 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.157+0000 7f5a3eb59640 1 Processor -- start 2024-04-15T15:17:45.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.157+0000 7f5a3eb59640 1 -- start start 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3eb59640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a3815e790 0x7f5a380d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3eb59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380d1610 0x7f5a380d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3eb59640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a380d2480 con 0x7f5a380cfd20 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3eb59640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f5a380d2db0 con 0x7f5a3815e790 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3eb59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5a380d2760 con 0x7f5a380d1610 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3c8f5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a3815e790 0x7f5a380d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a37fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5a380cfd20 0x7f5a380d0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52052/0 (socket says 172.21.15.46:52052) 2024-04-15T15:17:45.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3c8f5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5a3815e790 0x7f5a380d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40896/0 (socket says 172.21.15.46:40896) 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a37fff640 1 -- 172.21.15.46:0/210390048 learned_addr learned my addr 172.21.15.46:0/210390048 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3d0f6640 1 --2- 172.21.15.46:0/210390048 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380d1610 0x7f5a380d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3c8f5640 1 -- 172.21.15.46:0/210390048 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380d1610 msgr2=0x7f5a380d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3c8f5640 1 --2- 172.21.15.46:0/210390048 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380d1610 0x7f5a380d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3c8f5640 1 -- 172.21.15.46:0/210390048 >> v1:172.21.15.46:6789/0 conn(0x7f5a380cfd20 legacy=0x7f5a380d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a3c8f5640 1 -- 172.21.15.46:0/210390048 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a380d2a40 con 0x7f5a3815e790 2024-04-15T15:17:45.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.158+0000 7f5a377fe640 1 -- 172.21.15.46:0/210390048 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309439198 0 0) 0x7f5a380d2480 con 0x7f5a380cfd20 2024-04-15T15:17:45.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.159+0000 7f5a3c8f5640 1 --2- 172.21.15.46:0/210390048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a3815e790 0x7f5a380d3240 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5a380d2f30 tx=0x7f5a2c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e1da773c16c1b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.159+0000 7f5a377fe640 1 -- 172.21.15.46:0/210390048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c035020 con 0x7f5a3815e790 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.159+0000 7f5a377fe640 1 -- 172.21.15.46:0/210390048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a2c031800 con 0x7f5a3815e790 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.159+0000 7f5a377fe640 1 -- 172.21.15.46:0/210390048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c031bd0 con 0x7f5a3815e790 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 -- 172.21.15.46:0/210390048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a3815e790 msgr2=0x7f5a380d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/210390048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a3815e790 0x7f5a380d3240 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5a380d2f30 tx=0x7f5a2c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 -- 172.21.15.46:0/210390048 shutdown_connections 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/210390048 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380d1610 0x7f5a380d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/210390048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a3815e790 0x7f5a380d3240 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 -- 172.21.15.46:0/210390048 >> 172.21.15.46:0/210390048 conn(0x7f5a3808bf60 msgr2=0x7f5a380d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 -- 172.21.15.46:0/210390048 shutdown_connections 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.160+0000 7f5a3eb59640 1 -- 172.21.15.46:0/210390048 wait complete. 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.161+0000 7f5a3eb59640 1 Processor -- start 2024-04-15T15:17:45.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.161+0000 7f5a3eb59640 1 -- start start 2024-04-15T15:17:45.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380cfd20 0x7f5a38159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 0x7f5a38152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a3815e790 0x7f5a3815c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3815ecf0 con 0x7f5a380d1610 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3815eb70 con 0x7f5a3815e790 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f5a380b74f0 con 0x7f5a380cfd20 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 0x7f5a38152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 0x7f5a38152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40912/0 (socket says 172.21.15.46:40912) 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 -- 172.21.15.46:0/392558546 learned_addr learned my addr 172.21.15.46:0/392558546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3c8f5640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380cfd20 0x7f5a38159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3d0f6640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a3815e790 0x7f5a3815c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 -- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380cfd20 msgr2=0x7f5a38159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380cfd20 0x7f5a38159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 -- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a3815e790 msgr2=0x7f5a3815c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a3815e790 0x7f5a3815c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 -- 172.21.15.46:0/392558546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a38159a30 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3c8f5640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380cfd20 0x7f5a38159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3d0f6640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a3815e790 0x7f5a3815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a37fff640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 0x7f5a38152ce0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400c4d0 tx=0x7f5a2400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a24018070 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a24007500 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a38159d10 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a24004a20 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.162+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a3815a2d0 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.164+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a380689e0 con 0x7f5a380d1610 2024-04-15T15:17:45.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.164+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5a2401b070 con 0x7f5a380d1610 2024-04-15T15:17:45.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.164+0000 7f5a357fa640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5a1c06c9b0 0x7f5a1c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.165+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f5a2408b0a0 con 0x7f5a380d1610 2024-04-15T15:17:45.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.165+0000 7f5a3c8f5640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5a1c06c9b0 0x7f5a1c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.165+0000 7f5a3c8f5640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5a1c06c9b0 0x7f5a1c06ee70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c036000 tx=0x7f5a2c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.169+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a240592e0 con 0x7f5a380d1610 2024-04-15T15:17:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.352+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a380d1ce0 con 0x7f5a380d1610 2024-04-15T15:17:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.353+0000 7f5a357fa640 1 -- 172.21.15.46:0/392558546 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f5a24058be0 con 0x7f5a380d1610 2024-04-15T15:17:45.355 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.354+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5a1c06c9b0 msgr2=0x7f5a1c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.354+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5a1c06c9b0 0x7f5a1c06ee70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c036000 tx=0x7f5a2c008030 comp rx=0 tx=0).stop 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.354+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 msgr2=0x7f5a38152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.354+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 0x7f5a38152ce0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f5a2400c4d0 tx=0x7f5a2400c990 comp rx=0 tx=0).stop 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 shutdown_connections 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f5a1c06c9b0 0x7f5a1c06ee70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a3815e790 0x7f5a3815c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a380d1610 0x7f5a38152ce0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 --2- 172.21.15.46:0/392558546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f5a380cfd20 0x7f5a38159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 >> 172.21.15.46:0/392558546 conn(0x7f5a3808bf60 msgr2=0x7f5a38074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 shutdown_connections 2024-04-15T15:17:45.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.355+0000 7f5a3eb59640 1 -- 172.21.15.46:0/392558546 wait complete. 2024-04-15T15:17:45.368 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:45.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.470+0000 7f8347e35640 1 Processor -- start 2024-04-15T15:17:45.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f8347e35640 1 -- start start 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f8347e35640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f83400b3160 0x7f834015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f8347e35640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83400ae280 0x7f83400ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f8347e35640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834015cb50 con 0x7f83400d0690 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f8347e35640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f834015bfd0 con 0x7f83400ae280 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f8347e35640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f834015c2b0 con 0x7f83400b3160 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.471+0000 7f83463d2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83400ae280 0x7f83400ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83463d2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83400ae280 0x7f83400ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40914/0 (socket says 172.21.15.46:40914) 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83463d2640 1 -- 172.21.15.46:0/3614243302 learned_addr learned my addr 172.21.15.46:0/3614243302 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83453d0640 1 --2- 172.21.15.46:0/3614243302 >> v2:172.21.15.78:3300/0 conn(0x7f83400b3160 0x7f834015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83463d2640 1 -- 172.21.15.46:0/3614243302 >> v2:172.21.15.78:3300/0 conn(0x7f83400b3160 msgr2=0x7f834015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83463d2640 1 --2- 172.21.15.46:0/3614243302 >> v2:172.21.15.78:3300/0 conn(0x7f83400b3160 0x7f834015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83463d2640 1 -- 172.21.15.46:0/3614243302 >> v1:172.21.15.78:6789/0 conn(0x7f83400d0690 legacy=0x7f83400d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.472+0000 7f83463d2640 1 -- 172.21.15.46:0/3614243302 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834015c590 con 0x7f83400ae280 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f83463d2640 1 --2- 172.21.15.46:0/3614243302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400ae280 0x7f83400ae710 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f834006fec0 tx=0x7f833802db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4696c8000fef1f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8344bcf640 1 -- 172.21.15.46:0/3614243302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833802f8e0 con 0x7f83400ae280 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8344bcf640 1 -- 172.21.15.46:0/3614243302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833802fa80 con 0x7f83400ae280 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8344bcf640 1 -- 172.21.15.46:0/3614243302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833802fe50 con 0x7f83400ae280 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 -- 172.21.15.46:0/3614243302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400ae280 msgr2=0x7f83400ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 --2- 172.21.15.46:0/3614243302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400ae280 0x7f83400ae710 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f834006fec0 tx=0x7f833802db20 comp rx=0 tx=0).stop 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 -- 172.21.15.46:0/3614243302 shutdown_connections 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 --2- 172.21.15.46:0/3614243302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400ae280 0x7f83400ae710 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 --2- 172.21.15.46:0/3614243302 >> v2:172.21.15.78:3300/0 conn(0x7f83400b3160 0x7f834015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 -- 172.21.15.46:0/3614243302 >> 172.21.15.46:0/3614243302 conn(0x7f834006be00 msgr2=0x7f834006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 -- 172.21.15.46:0/3614243302 shutdown_connections 2024-04-15T15:17:45.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.473+0000 7f8347e35640 1 -- 172.21.15.46:0/3614243302 wait complete. 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 Processor -- start 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 -- start start 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400ae280 0x7f83400b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 0x7f83401586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83400d0690 0x7f8340158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f834015bbd0 con 0x7f83400b3160 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f83400b4fb0 con 0x7f83400ae280 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f8347e35640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f834015bd50 con 0x7f83400d0690 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f83453d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 0x7f83401586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f83453d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 0x7f83401586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40920/0 (socket says 172.21.15.46:40920) 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.475+0000 7f83453d0640 1 -- 172.21.15.46:0/3552170320 learned_addr learned my addr 172.21.15.46:0/3552170320 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f8345bd1640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400ae280 0x7f83400b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83463d2640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83400d0690 0x7f8340158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83453d0640 1 -- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83400d0690 msgr2=0x7f8340158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83453d0640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83400d0690 0x7f8340158be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83453d0640 1 -- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400ae280 msgr2=0x7f83400b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83453d0640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400ae280 0x7f83400b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83453d0640 1 -- 172.21.15.46:0/3552170320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83400b1b40 con 0x7f83400b3160 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f83453d0640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 0x7f83401586a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f833400d5f0 tx=0x7f833400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83340125f0 con 0x7f83400b3160 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8334019070 con 0x7f83400b3160 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834015ddd0 con 0x7f83400b3160 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83340129d0 con 0x7f83400b3160 2024-04-15T15:17:45.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.476+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f834015e390 con 0x7f83400b3160 2024-04-15T15:17:45.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.477+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83400689e0 con 0x7f83400b3160 2024-04-15T15:17:45.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.478+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8334012b70 con 0x7f83400b3160 2024-04-15T15:17:45.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.478+0000 7f833e7fc640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f832c06c9b0 0x7f832c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.478+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f833408c160 con 0x7f83400b3160 2024-04-15T15:17:45.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.479+0000 7f8345bd1640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f832c06c9b0 0x7f832c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.479+0000 7f8345bd1640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f832c06c9b0 0x7f832c06ee70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8330009f20 tx=0x7f83300094d0 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.482+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f833405a3a0 con 0x7f83400b3160 2024-04-15T15:17:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.704+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83400739e0 con 0x7f83400b3160 2024-04-15T15:17:45.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.705+0000 7f833e7fc640 1 -- 172.21.15.46:0/3552170320 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5176 (secure 0 0 0) 0x7f8334059ca0 con 0x7f83400b3160 2024-04-15T15:17:45.707 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:45.707 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:34.636332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5946,"qdb_cluster":[5946]},"id":2}]} 2024-04-15T15:17:45.707 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f832c06c9b0 msgr2=0x7f832c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f832c06c9b0 0x7f832c06ee70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8330009f20 tx=0x7f83300094d0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 msgr2=0x7f83401586a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 0x7f83401586a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f833400d5f0 tx=0x7f833400dab0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 shutdown_connections 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f832c06c9b0 0x7f832c06ee70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f83400d0690 0x7f8340158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83400b3160 0x7f83401586a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 --2- 172.21.15.46:0/3552170320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83400ae280 0x7f83400b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 >> 172.21.15.46:0/3552170320 conn(0x7f834006be00 msgr2=0x7f83400cf340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 shutdown_connections 2024-04-15T15:17:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.707+0000 7f8347e35640 1 -- 172.21.15.46:0/3552170320 wait complete. 2024-04-15T15:17:45.720 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 Processor -- start 2024-04-15T15:17:45.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 -- start start 2024-04-15T15:17:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 0x7f7c2c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7c2c0b4350 0x7f7c2c0b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2c0b10d0 con 0x7f7c2c0b1b60 2024-04-15T15:17:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f7c2c15f090 con 0x7f7c2c0b4350 2024-04-15T15:17:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.824+0000 7f7c314b1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0cdf80 con 0x7f7c2c15ecb0 2024-04-15T15:17:45.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7c2c0b1b60 0x7f7c2c0ce410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52054/0 (socket says 172.21.15.46:52054) 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2b7fe640 1 -- 172.21.15.46:0/2779272141 learned_addr learned my addr 172.21.15.46:0/2779272141 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152959170 0 0) 0x7f7c2c0b10d0 con 0x7f7c2c0b1b60 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c18003520 con 0x7f7c2c0b1b60 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2affd640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 0x7f7c2c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2a7fc640 1 --2- 172.21.15.46:0/2779272141 >> v2:172.21.15.78:3300/0 conn(0x7f7c2c0b4350 0x7f7c2c0b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3086502723 0 0) 0x7f7c18003520 con 0x7f7c2c0b1b60 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 msgr2=0x7f7c2c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 0x7f7c2c0b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 >> v2:172.21.15.78:3300/0 conn(0x7f7c2c0b4350 msgr2=0x7f7c2c0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 --2- 172.21.15.46:0/2779272141 >> v2:172.21.15.78:3300/0 conn(0x7f7c2c0b4350 0x7f7c2c0b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c0af920 con 0x7f7c2c0b1b60 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f7c24001670 con 0x7f7c2c0b1b60 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2a7fc640 1 --2- 172.21.15.46:0/2779272141 >> v2:172.21.15.78:3300/0 conn(0x7f7c2c0b4350 0x7f7c2c0b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:45.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 >> v1:172.21.15.46:6789/0 conn(0x7f7c2c0b1b60 legacy=0x7f7c2c0ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 0x7f7c18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c18004d40 0x7f7c180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c15f090 con 0x7f7c18004d40 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0cdf80 con 0x7f7c180039b0 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7c1800ba80 con 0x7f7c2c15ecb0 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2b7fe640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c18004d40 0x7f7c180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2affd640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.825+0000 7f7c2a7fc640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 0x7f7c18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c2b7fe640 1 -- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 msgr2=0x7f7c18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c2b7fe640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c15ecb0 0x7f7c18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c2b7fe640 1 -- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c180039b0 msgr2=0x7f7c18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c2b7fe640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c180039b0 0x7f7c18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c2b7fe640 1 -- 172.21.15.46:0/2779272141 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c18003520 con 0x7f7c18004d40 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c2b7fe640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c18004d40 0x7f7c180051c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7c24007fd0 tx=0x7f7c18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f27a13140f108aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24007d50 con 0x7f7c18004d40 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c24030070 con 0x7f7c18004d40 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.826+0000 7f7c29ffb640 1 -- 172.21.15.46:0/2779272141 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2402cd40 con 0x7f7c18004d40 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c314b1640 1 -- 172.21.15.46:0/2779272141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c18004d40 msgr2=0x7f7c180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2779272141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c18004d40 0x7f7c180051c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7c24007fd0 tx=0x7f7c18005830 comp rx=0 tx=0).stop 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c2b7fe640 1 -- 172.21.15.46:0/2779272141 reap_dead start 2024-04-15T15:17:45.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c314b1640 1 -- 172.21.15.46:0/2779272141 shutdown_connections 2024-04-15T15:17:45.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c314b1640 1 -- 172.21.15.46:0/2779272141 >> 172.21.15.46:0/2779272141 conn(0x7f7c2c0b6870 msgr2=0x7f7c2c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:45.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c314b1640 1 -- 172.21.15.46:0/2779272141 shutdown_connections 2024-04-15T15:17:45.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.827+0000 7f7c314b1640 1 -- 172.21.15.46:0/2779272141 wait complete. 2024-04-15T15:17:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.828+0000 7f7c314b1640 1 Processor -- start 2024-04-15T15:17:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.828+0000 7f7c314b1640 1 -- start start 2024-04-15T15:17:45.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c314b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 0x7f7c2c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c314b1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c2c1615b0 0x7f7c2c0cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c314b1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c0cc180 0x7f7c2c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c314b1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b4790 con 0x7f7c2c14eec0 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c314b1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2c0afee0 con 0x7f7c2c1615b0 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c314b1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7c2c0b4910 con 0x7f7c2c0cc180 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 0x7f7c2c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 0x7f7c2c1591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40932/0 (socket says 172.21.15.46:40932) 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 -- 172.21.15.46:0/2697940137 learned_addr learned my addr 172.21.15.46:0/2697940137 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2a7fc640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c2c1615b0 0x7f7c2c0cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2b7fe640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c0cc180 0x7f7c2c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 -- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c0cc180 msgr2=0x7f7c2c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c0cc180 0x7f7c2c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 -- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c2c1615b0 msgr2=0x7f7c2c0cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c2c1615b0 0x7f7c2c0cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:45.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.829+0000 7f7c2affd640 1 -- 172.21.15.46:0/2697940137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2c14f2a0 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.830+0000 7f7c2affd640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 0x7f7c2c1591e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009fc0 tx=0x7f7c1c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.830+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c015430 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.830+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1c0155d0 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.830+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2c158620 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.830+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1c0159c0 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.830+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2c15f3b0 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.831+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2c0689e0 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.831+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7c1c015b60 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.832+0000 7f7c237fe640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7c0c06c9b0 0x7f7c0c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.832+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f7c1c08cf40 con 0x7f7c2c14eec0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.832+0000 7f7c2a7fc640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7c0c06c9b0 0x7f7c0c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:45.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.833+0000 7f7c2a7fc640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7c0c06c9b0 0x7f7c0c06ee70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0708c0 tx=0x7f7c14008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:45.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:45.836+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c1c019030 con 0x7f7c2c14eec0 2024-04-15T15:17:46.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.022+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c2c0b0470 con 0x7f7c2c14eec0 2024-04-15T15:17:46.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.023+0000 7f7c237fe640 1 -- 172.21.15.46:0/2697940137 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7f7c1c010040 con 0x7f7c2c14eec0 2024-04-15T15:17:46.025 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:46.025 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7c0c06c9b0 msgr2=0x7f7c0c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7c0c06c9b0 0x7f7c0c06ee70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c0708c0 tx=0x7f7c14008040 comp rx=0 tx=0).stop 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 msgr2=0x7f7c2c1591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 0x7f7c2c1591e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c009fc0 tx=0x7f7c1c00f580 comp rx=0 tx=0).stop 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 shutdown_connections 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7c0c06c9b0 0x7f7c0c06ee70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7c2c0cc180 0x7f7c2c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c2c1615b0 0x7f7c2c0cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 --2- 172.21.15.46:0/2697940137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2c14eec0 0x7f7c2c1591e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 >> 172.21.15.46:0/2697940137 conn(0x7f7c2c0b6870 msgr2=0x7f7c2c0ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 shutdown_connections 2024-04-15T15:17:46.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.025+0000 7f7c314b1640 1 -- 172.21.15.46:0/2697940137 wait complete. 2024-04-15T15:17:46.040 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:17:46.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.143+0000 7f6dfa6a5640 1 Processor -- start 2024-04-15T15:17:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.143+0000 7f6dfa6a5640 1 -- start start 2024-04-15T15:17:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6dfa6a5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f6df409a9d0 0x7f6df4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6dfa6a5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df4070720 con 0x7f6df40701f0 2024-04-15T15:17:46.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6dfa6a5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df4070a90 con 0x7f6df415fb30 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6dfa6a5640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f6df409ba90 con 0x7f6df409a9d0 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6df40701f0 0x7f6df409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52056/0 (socket says 172.21.15.46:52056) 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df37fe640 1 -- 172.21.15.46:0/108228610 learned_addr learned my addr 172.21.15.46:0/108228610 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831011288 0 0) 0x7f6df4070720 con 0x7f6df40701f0 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ddc003520 con 0x7f6df40701f0 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df3fff640 1 --2- 172.21.15.46:0/108228610 >> v2:172.21.15.78:3300/0 conn(0x7f6df409a9d0 0x7f6df4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 500017891 0 0) 0x7f6ddc003520 con 0x7f6df40701f0 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 >> v2:172.21.15.78:3300/0 conn(0x7f6df409a9d0 msgr2=0x7f6df4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:46.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/108228610 >> v2:172.21.15.78:3300/0 conn(0x7f6df409a9d0 0x7f6df4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 >> v1:172.21.15.78:6789/0 conn(0x7f6df415fb30 legacy=0x7f6df40d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df4070d70 con 0x7f6df40701f0 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f6de8001670 con 0x7f6df40701f0 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.144+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 >> v1:172.21.15.46:6789/0 conn(0x7f6df40701f0 legacy=0x7f6df409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6ddc003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ddc004800 0x7f6ddc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df2ffd640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6ddc0051c0 0x7f6ddc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc00c0c0 con 0x7f6ddc0039b0 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc00c270 con 0x7f6ddc004800 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc00c470 con 0x7f6ddc0051c0 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df3fff640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6ddc003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df37fe640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ddc004800 0x7f6ddc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df8c42640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6ddc0051c0 0x7f6ddc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df3fff640 1 -- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6ddc0051c0 msgr2=0x7f6ddc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df3fff640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6ddc0051c0 0x7f6ddc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df3fff640 1 -- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ddc004800 msgr2=0x7f6ddc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df3fff640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ddc004800 0x7f6ddc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df3fff640 1 -- 172.21.15.46:0/108228610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc003520 con 0x7f6ddc0039b0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.145+0000 7f6df8c42640 1 -- 172.21.15.46:0/108228610 reap_dead start 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.146+0000 7f6df3fff640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6ddc003db0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6de400ba00 tx=0x7f6ddc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3da432eada7de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.146+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4036020 con 0x7f6ddc0039b0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.146+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de40325f0 con 0x7f6ddc0039b0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.146+0000 7f6df2ffd640 1 -- 172.21.15.46:0/108228610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de400b600 con 0x7f6ddc0039b0 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/108228610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 msgr2=0x7f6ddc003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6ddc003db0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6de400ba00 tx=0x7f6ddc005830 comp rx=0 tx=0).stop 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/108228610 shutdown_connections 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/108228610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6ddc003db0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/108228610 >> 172.21.15.46:0/108228610 conn(0x7f6df40d2460 msgr2=0x7f6df40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/108228610 shutdown_connections 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.147+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/108228610 wait complete. 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.148+0000 7f6dfa6a5640 1 Processor -- start 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.148+0000 7f6dfa6a5640 1 -- start start 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6dfa6a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6df40b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6dfa6a5640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df40b6af0 0x7f6df4155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6dfa6a5640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6df40c7390 0x7f6df40b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6dfa6a5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6df415ffc0 con 0x7f6ddc0039b0 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6dfa6a5640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f6df40d08c0 con 0x7f6df40b6af0 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6dfa6a5640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f6df40b1c60 con 0x7f6df40c7390 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6df40b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6df40b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40948/0 (socket says 172.21.15.46:40948) 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 -- 172.21.15.46:0/2616837570 learned_addr learned my addr 172.21.15.46:0/2616837570 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df37fe640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df40b6af0 0x7f6df4155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df8c42640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6df40c7390 0x7f6df40b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 -- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6df40c7390 msgr2=0x7f6df40b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6df40c7390 0x7f6df40b7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 -- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df40b6af0 msgr2=0x7f6df4155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df40b6af0 0x7f6df4155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 -- 172.21.15.46:0/2616837570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df40b57d0 con 0x7f6ddc0039b0 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.149+0000 7f6df3fff640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6df40b5290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6de403b670 tx=0x7f6de4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.150+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4042070 con 0x7f6ddc0039b0 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.150+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de403ba60 con 0x7f6ddc0039b0 2024-04-15T15:17:46.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.150+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df40b5a50 con 0x7f6ddc0039b0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.150+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4036040 con 0x7f6ddc0039b0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.150+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df40b6010 con 0x7f6ddc0039b0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.151+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df40689e0 con 0x7f6ddc0039b0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.151+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6de404d060 con 0x7f6ddc0039b0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.152+0000 7f6df0ff9640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6de006c980 0x7f6de006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.152+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f6de40add40 con 0x7f6ddc0039b0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.152+0000 7f6df37fe640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6de006c980 0x7f6de006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.152+0000 7f6df37fe640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6de006c980 0x7f6de006ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6df4161da0 tx=0x7f6de8009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.156+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6df40b57d0 con 0x7f6ddc0039b0 2024-04-15T15:17:46.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.382+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6df4070720 con 0x7f6ddc0039b0 2024-04-15T15:17:46.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.382+0000 7f6df0ff9640 1 -- 172.21.15.46:0/2616837570 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5176 (secure 0 0 0) 0x7f6de402d040 con 0x7f6ddc0039b0 2024-04-15T15:17:46.384 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:46.384 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:34.636332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5946,"qdb_cluster":[5946]},"id":2}]} 2024-04-15T15:17:46.384 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6de006c980 msgr2=0x7f6de006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6de006c980 0x7f6de006ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6df4161da0 tx=0x7f6de8009040 comp rx=0 tx=0).stop 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 msgr2=0x7f6df40b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6df40b5290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6de403b670 tx=0x7f6de4004730 comp rx=0 tx=0).stop 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 shutdown_connections 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f6de006c980 0x7f6de006ee40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f6df40c7390 0x7f6df40b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df40b6af0 0x7f6df4155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 --2- 172.21.15.46:0/2616837570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ddc0039b0 0x7f6df40b5290 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 >> 172.21.15.46:0/2616837570 conn(0x7f6df40d2460 msgr2=0x7f6df40b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:46.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 shutdown_connections 2024-04-15T15:17:46.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.384+0000 7f6dfa6a5640 1 -- 172.21.15.46:0/2616837570 wait complete. 2024-04-15T15:17:46.398 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:46.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.500+0000 7fbec03d4640 1 Processor -- start 2024-04-15T15:17:46.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.500+0000 7fbec03d4640 1 -- start start 2024-04-15T15:17:46.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbec03d4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbeb809a9d0 0x7fbeb8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbec03d4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbec03d4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeb8070720 con 0x7fbeb809adb0 2024-04-15T15:17:46.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbec03d4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fbeb809ba90 con 0x7fbeb809a9d0 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbec03d4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb8160070 con 0x7fbeb80701f0 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe170640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbeb809a9d0 0x7fbeb8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe971640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbeb809adb0 0x7fbeb80d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52072/0 (socket says 172.21.15.46:52072) 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe971640 1 -- 172.21.15.46:0/2054316367 learned_addr learned my addr 172.21.15.46:0/2054316367 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebd96f640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe170640 1 -- 172.21.15.46:0/2054316367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 msgr2=0x7fbeb815fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe170640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe170640 1 -- 172.21.15.46:0/2054316367 >> v1:172.21.15.46:6789/0 conn(0x7fbeb809adb0 legacy=0x7fbeb80d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebd16e640 1 -- 172.21.15.46:0/2054316367 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3913138474 0 0) 0x7fbeb8070720 con 0x7fbeb809adb0 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebe170640 1 -- 172.21.15.46:0/2054316367 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb8070a90 con 0x7fbeb809a9d0 2024-04-15T15:17:46.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.501+0000 7fbebd96f640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.502+0000 7fbebe170640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb8160420 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40097d0 tx=0x7fbeb402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0d9517453a89e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.502+0000 7fbebd16e640 1 -- 172.21.15.46:0/2054316367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4035020 con 0x7fbeb809a9d0 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.502+0000 7fbebd16e640 1 -- 172.21.15.46:0/2054316367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb4031800 con 0x7fbeb809a9d0 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.502+0000 7fbebd16e640 1 -- 172.21.15.46:0/2054316367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb4031bd0 con 0x7fbeb809a9d0 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 -- 172.21.15.46:0/2054316367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 msgr2=0x7fbeb8160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb8160420 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbeb40097d0 tx=0x7fbeb402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 -- 172.21.15.46:0/2054316367 shutdown_connections 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2054316367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb8160420 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 -- 172.21.15.46:0/2054316367 >> 172.21.15.46:0/2054316367 conn(0x7fbeb80d2460 msgr2=0x7fbeb80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 -- 172.21.15.46:0/2054316367 shutdown_connections 2024-04-15T15:17:46.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.503+0000 7fbec03d4640 1 -- 172.21.15.46:0/2054316367 wait complete. 2024-04-15T15:17:46.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.504+0000 7fbec03d4640 1 Processor -- start 2024-04-15T15:17:46.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.504+0000 7fbec03d4640 1 -- start start 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbec03d4640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbec03d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbec03d4640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbeb809adb0 0x7fbeb81592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbec03d4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb8071050 con 0x7fbeb809a9d0 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbec03d4640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb80d08c0 con 0x7fbeb809adb0 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbec03d4640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb80b1c60 con 0x7fbeb80701f0 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb814a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40956/0 (socket says 172.21.15.46:40956) 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 -- 172.21.15.46:0/2289016957 learned_addr learned my addr 172.21.15.46:0/2289016957 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebe170640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebe971640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbeb809adb0 0x7fbeb81592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 -- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 msgr2=0x7fbeb815c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 -- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbeb809adb0 msgr2=0x7fbeb81592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbeb809adb0 0x7fbeb81592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 -- 172.21.15.46:0/2289016957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb814adc0 con 0x7fbeb809a9d0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebe971640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbeb809adb0 0x7fbeb81592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.505+0000 7fbebd96f640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb814a880 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbea800d800 tx=0x7fbea800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.506+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea8004030 con 0x7fbeb809a9d0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.506+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea80041d0 con 0x7fbeb809a9d0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.506+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb8159c40 con 0x7fbeb809a9d0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.506+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea80096a0 con 0x7fbeb809a9d0 2024-04-15T15:17:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.506+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb815a200 con 0x7fbeb809a9d0 2024-04-15T15:17:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.507+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb8071f70 con 0x7fbeb809a9d0 2024-04-15T15:17:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.507+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fbea8009840 con 0x7fbeb809a9d0 2024-04-15T15:17:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.508+0000 7fbeaeffd640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbe9c06c980 0x7fbe9c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.508+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fbea808bf80 con 0x7fbeb809a9d0 2024-04-15T15:17:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.508+0000 7fbebe170640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbe9c06c980 0x7fbe9c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.509+0000 7fbebe170640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbe9c06c980 0x7fbe9c06ee40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4036000 tx=0x7fbeb4008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.512+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbeb8071f70 con 0x7fbeb809a9d0 2024-04-15T15:17:46.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.735+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbeb80689e0 con 0x7fbeb809a9d0 2024-04-15T15:17:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.736+0000 7fbeaeffd640 1 -- 172.21.15.46:0/2289016957 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5176 (secure 0 0 0) 0x7fbeb814adc0 con 0x7fbeb809a9d0 2024-04-15T15:17:46.738 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:46.738 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:17:00.418468+0000","modified":"2024-04-15T15:17:34.636332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.46:6835/245964640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":245964640},{"type":"v1","addr":"172.21.15.46:6835","nonce":245964640}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5946,"qdb_cluster":[5946]},"id":2}]} 2024-04-15T15:17:46.738 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-15T15:17:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbe9c06c980 msgr2=0x7fbe9c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbe9c06c980 0x7fbe9c06ee40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4036000 tx=0x7fbeb4008030 comp rx=0 tx=0).stop 2024-04-15T15:17:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 msgr2=0x7fbeb814a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb814a880 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbea800d800 tx=0x7fbea800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 shutdown_connections 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fbe9c06c980 0x7fbe9c06ee40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbeb809adb0 0x7fbeb81592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbeb809a9d0 0x7fbeb814a880 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 --2- 172.21.15.46:0/2289016957 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fbeb80701f0 0x7fbeb815c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 >> 172.21.15.46:0/2289016957 conn(0x7fbeb80d2460 msgr2=0x7fbeb80ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 shutdown_connections 2024-04-15T15:17:46.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.738+0000 7fbec03d4640 1 -- 172.21.15.46:0/2289016957 wait complete. 2024-04-15T15:17:46.752 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.855+0000 7f9769e7b640 1 Processor -- start 2024-04-15T15:17:46.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.855+0000 7f9769e7b640 1 -- start start 2024-04-15T15:17:46.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9769e7b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97640cd000 0x7f97640d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9769e7b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f976409ab40 0x7f976409afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9769e7b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f976415dbd0 con 0x7f97641602d0 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9769e7b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f97640cd460 con 0x7f97640cd000 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9769e7b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f97640cd6a0 con 0x7f976409ab40 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9762ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f97640cd000 0x7f97640d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f97637fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97641602d0 0x7f97641606b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52088/0 (socket says 172.21.15.46:52088) 2024-04-15T15:17:46.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f97637fe640 1 -- 172.21.15.46:0/3480011345 learned_addr learned my addr 172.21.15.46:0/3480011345 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9763fff640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f976409ab40 0x7f976409afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.856+0000 7f9762ffd640 1 -- 172.21.15.46:0/3480011345 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f976409ab40 msgr2=0x7f976409afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f9762ffd640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f976409ab40 0x7f976409afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f9762ffd640 1 -- 172.21.15.46:0/3480011345 >> v1:172.21.15.46:6789/0 conn(0x7f97641602d0 legacy=0x7f97641606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f9762ffd640 1 -- 172.21.15.46:0/3480011345 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97640cd980 con 0x7f97640cd000 2024-04-15T15:17:46.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f9763fff640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f976409ab40 0x7f976409afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f97627fc640 1 -- 172.21.15.46:0/3480011345 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2080783004 0 0) 0x7f976415dbd0 con 0x7f97641602d0 2024-04-15T15:17:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f9762ffd640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97640cd000 0x7f97640d0ef0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f97580097a0 tx=0x7f975802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f193fa5e0de8431b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f97627fc640 1 -- 172.21.15.46:0/3480011345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758035020 con 0x7f97640cd000 2024-04-15T15:17:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f97627fc640 1 -- 172.21.15.46:0/3480011345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9758031830 con 0x7f97640cd000 2024-04-15T15:17:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.857+0000 7f97627fc640 1 -- 172.21.15.46:0/3480011345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758031c00 con 0x7f97640cd000 2024-04-15T15:17:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 -- 172.21.15.46:0/3480011345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97640cd000 msgr2=0x7f97640d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97640cd000 0x7f97640d0ef0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f97580097a0 tx=0x7f975802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 -- 172.21.15.46:0/3480011345 shutdown_connections 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97640cd000 0x7f97640d0ef0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 --2- 172.21.15.46:0/3480011345 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f976409ab40 0x7f976409afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 -- 172.21.15.46:0/3480011345 >> 172.21.15.46:0/3480011345 conn(0x7f97640d2460 msgr2=0x7f97640d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 -- 172.21.15.46:0/3480011345 shutdown_connections 2024-04-15T15:17:46.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.858+0000 7f9769e7b640 1 -- 172.21.15.46:0/3480011345 wait complete. 2024-04-15T15:17:46.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.859+0000 7f9769e7b640 1 Processor -- start 2024-04-15T15:17:46.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.859+0000 7f9769e7b640 1 -- start start 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9769e7b640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f976409ab40 0x7f97640b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9769e7b640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f97640cd000 0x7f976414bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9769e7b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 0x7f97640b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9769e7b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f97640ce220 con 0x7f97641602d0 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9769e7b640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f97640cdf40 con 0x7f976409ab40 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9769e7b640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f97640d1430 con 0x7f97640cd000 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9763fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 0x7f97640b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9763fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 0x7f97640b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40972/0 (socket says 172.21.15.46:40972) 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.860+0000 7f9763fff640 1 -- 172.21.15.46:0/2153113124 learned_addr learned my addr 172.21.15.46:0/2153113124 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9762ffd640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f97640cd000 0x7f976414bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f97637fe640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f976409ab40 0x7f97640b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9763fff640 1 -- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f97640cd000 msgr2=0x7f976414bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9763fff640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f97640cd000 0x7f976414bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9763fff640 1 -- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f976409ab40 msgr2=0x7f97640b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9763fff640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f976409ab40 0x7f97640b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9763fff640 1 -- 172.21.15.46:0/2153113124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97640b5580 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9762ffd640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f97640cd000 0x7f976414bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f97637fe640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f976409ab40 0x7f97640b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9763fff640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 0x7f97640b78d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f975c007df0 tx=0x7f975c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c01a070 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975c003990 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976414c4e0 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975c003d80 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.861+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97640b70d0 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.862+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97640689e0 con 0x7f97641602d0 2024-04-15T15:17:46.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.863+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f975c01d070 con 0x7f97641602d0 2024-04-15T15:17:46.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.863+0000 7f974bfff640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f975006c980 0x7f975006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:46.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.863+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f975c08c000 con 0x7f97641602d0 2024-04-15T15:17:46.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.863+0000 7f97637fe640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f975006c980 0x7f975006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:46.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.864+0000 7f97637fe640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f975006c980 0x7f975006ee40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9754000f60 tx=0x7f9754009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:46.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:46.867+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f975c010070 con 0x7f97641602d0 2024-04-15T15:17:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.051+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97640b4e20 con 0x7f97641602d0 2024-04-15T15:17:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.053+0000 7f974bfff640 1 -- 172.21.15.46:0/2153113124 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17070 (secure 0 0 0) 0x7f975c05a1c0 con 0x7f97641602d0 2024-04-15T15:17:47.055 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:47.055 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":42,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:30.625425+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:58.861526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:59.361892+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.054+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f975006c980 msgr2=0x7f975006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.054+0000 7f9769e7b640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f975006c980 0x7f975006ee40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9754000f60 tx=0x7f9754009040 comp rx=0 tx=0).stop 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.054+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 msgr2=0x7f97640b78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.054+0000 7f9769e7b640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 0x7f97640b78d0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f975c007df0 tx=0x7f975c00d4a0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 shutdown_connections 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f975006c980 0x7f975006ee40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97641602d0 0x7f97640b78d0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f97640cd000 0x7f976414bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 --2- 172.21.15.46:0/2153113124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f976409ab40 0x7f97640b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 >> 172.21.15.46:0/2153113124 conn(0x7f97640d2460 msgr2=0x7f97641598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 shutdown_connections 2024-04-15T15:17:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.055+0000 7f9769e7b640 1 -- 172.21.15.46:0/2153113124 wait complete. 2024-04-15T15:17:47.068 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:17:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.172+0000 7fd126b59640 1 Processor -- start 2024-04-15T15:17:47.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.172+0000 7fd126b59640 1 -- start start 2024-04-15T15:17:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd126b59640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd1200ce8e0 0x7fd120070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd126b59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071d90 0x7fd12009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd126b59640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd12006ff30 con 0x7fd120071240 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd126b59640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd120070820 con 0x7fd1200ce8e0 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd126b59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd1200cecc0 con 0x7fd120071d90 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd1248f5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd1200ce8e0 0x7fd120070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071d90 0x7fd12009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd1248f5640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd1200ce8e0 0x7fd120070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:50226/0 (socket says 172.21.15.46:50226) 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.173+0000 7fd1248f5640 1 -- 172.21.15.46:0/4126793681 learned_addr learned my addr 172.21.15.46:0/4126793681 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.174+0000 7fd1248f5640 1 -- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071d90 msgr2=0x7fd12009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.174+0000 7fd1248f5640 1 --2- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071d90 0x7fd12009a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.174+0000 7fd1248f5640 1 -- 172.21.15.46:0/4126793681 >> v1:172.21.15.78:6789/0 conn(0x7fd120071240 legacy=0x7fd120160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.174+0000 7fd1248f5640 1 -- 172.21.15.46:0/4126793681 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12009b9d0 con 0x7fd1200ce8e0 2024-04-15T15:17:47.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.174+0000 7fd11ffff640 1 --2- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071d90 0x7fd12009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.174+0000 7fd1248f5640 1 --2- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd120070cb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd110009870 tx=0x7fd11002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9303c128eaf03332 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd11f7fe640 1 -- 172.21.15.46:0/4126793681 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd110035020 con 0x7fd1200ce8e0 2024-04-15T15:17:47.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd11f7fe640 1 -- 172.21.15.46:0/4126793681 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd110031640 con 0x7fd1200ce8e0 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd11f7fe640 1 -- 172.21.15.46:0/4126793681 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11002d040 con 0x7fd1200ce8e0 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 -- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 msgr2=0x7fd120070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 --2- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd120070cb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd110009870 tx=0x7fd11002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 -- 172.21.15.46:0/4126793681 shutdown_connections 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 --2- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071d90 0x7fd12009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 --2- 172.21.15.46:0/4126793681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd120070cb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 -- 172.21.15.46:0/4126793681 >> 172.21.15.46:0/4126793681 conn(0x7fd1200b3f40 msgr2=0x7fd1200b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 -- 172.21.15.46:0/4126793681 shutdown_connections 2024-04-15T15:17:47.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.175+0000 7fd126b59640 1 -- 172.21.15.46:0/4126793681 wait complete. 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.176+0000 7fd126b59640 1 Processor -- start 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.176+0000 7fd126b59640 1 -- start start 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd126b59640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071240 0x7fd1200b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd126b59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 0x7fd12015f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd126b59640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd1201f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd126b59640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1200704b0 con 0x7fd120071d90 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd126b59640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd1200701d0 con 0x7fd1200ce8e0 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd126b59640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd120071970 con 0x7fd120071240 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 0x7fd12015f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd11ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 0x7fd12015f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40986/0 (socket says 172.21.15.46:40986) 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.177+0000 7fd11ffff640 1 -- 172.21.15.46:0/1401820697 learned_addr learned my addr 172.21.15.46:0/1401820697 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd1248f5640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071240 0x7fd1200b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd1250f6640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd1201f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11ffff640 1 -- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071240 msgr2=0x7fd1200b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11ffff640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071240 0x7fd1200b78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11ffff640 1 -- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 msgr2=0x7fd1201f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11ffff640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd1201f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11ffff640 1 -- 172.21.15.46:0/1401820697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd120155f70 con 0x7fd120071d90 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11ffff640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 0x7fd12015f4d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e960 tx=0x7fd10800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd108017070 con 0x7fd120071d90 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd10800ca00 con 0x7fd120071d90 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1200b6840 con 0x7fd120071d90 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10800cdd0 con 0x7fd120071d90 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.178+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1200b6e00 con 0x7fd120071d90 2024-04-15T15:17:47.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.179+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1200689e0 con 0x7fd120071d90 2024-04-15T15:17:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.180+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd10801a070 con 0x7fd120071d90 2024-04-15T15:17:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.180+0000 7fd11d7fa640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd10006c980 0x7fd10006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.180+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd10808bc90 con 0x7fd120071d90 2024-04-15T15:17:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.180+0000 7fd1248f5640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd10006c980 0x7fd10006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.181+0000 7fd1248f5640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd10006c980 0x7fd10006ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd110036000 tx=0x7fd110008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:47.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.184+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd108059ed0 con 0x7fd120071d90 2024-04-15T15:17:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.409+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd1200b4ca0 con 0x7fd120071d90 2024-04-15T15:17:47.411 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-04-15T15:17:47.410+0000 7fb7c20f9640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:17:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.411+0000 7fd11d7fa640 1 -- 172.21.15.46:0/1401820697 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7fd1080597d0 con 0x7fd120071d90 2024-04-15T15:17:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd10006c980 msgr2=0x7fd10006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd10006c980 0x7fd10006ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd110036000 tx=0x7fd110008030 comp rx=0 tx=0).stop 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 msgr2=0x7fd12015f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 0x7fd12015f4d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd10800e960 tx=0x7fd10800ee20 comp rx=0 tx=0).stop 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 shutdown_connections 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd10006c980 0x7fd10006ee40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1200ce8e0 0x7fd1201f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120071d90 0x7fd12015f4d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 --2- 172.21.15.46:0/1401820697 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd120071240 0x7fd1200b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 >> 172.21.15.46:0/1401820697 conn(0x7fd1200b3f40 msgr2=0x7fd1200af9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:47.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 shutdown_connections 2024-04-15T15:17:47.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.414+0000 7fd126b59640 1 -- 172.21.15.46:0/1401820697 wait complete. 2024-04-15T15:17:47.419 INFO:tasks.ceph.mds.c.smithi046.stderr: -1374> 2024-04-15T15:17:36.637+0000 7f41e34a2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:17:47.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:17:47.446 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2024-04-15T15:17:47.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 Processor -- start 2024-04-15T15:17:47.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 -- start start 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8df40d2360 0x7f8df40d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f8df415eae0 0x7f8df40d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df40b47a0 con 0x7f8df415e700 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f8df40b53b0 con 0x7f8df40d2360 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.533+0000 7f8dfa277640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f8df40b4a80 con 0x7f8df415eae0 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8df40d2360 0x7f8df40d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8df415e700 0x7f8df40b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52100/0 (socket says 172.21.15.46:52100) 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df3fff640 1 -- 172.21.15.46:0/1047477933 learned_addr learned my addr 172.21.15.46:0/1047477933 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df8814640 1 --2- 172.21.15.46:0/1047477933 >> v2:172.21.15.78:3300/0 conn(0x7f8df415eae0 0x7f8df40d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df37fe640 1 -- 172.21.15.46:0/1047477933 >> v2:172.21.15.78:3300/0 conn(0x7f8df415eae0 msgr2=0x7f8df40d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df37fe640 1 --2- 172.21.15.46:0/1047477933 >> v2:172.21.15.78:3300/0 conn(0x7f8df415eae0 0x7f8df40d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df37fe640 1 -- 172.21.15.46:0/1047477933 >> v1:172.21.15.46:6789/0 conn(0x7f8df415e700 legacy=0x7f8df40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df37fe640 1 -- 172.21.15.46:0/1047477933 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df40b4d60 con 0x7f8df40d2360 2024-04-15T15:17:47.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.534+0000 7f8df2ffd640 1 -- 172.21.15.46:0/1047477933 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005329112 0 0) 0x7f8df40b47a0 con 0x7f8df415e700 2024-04-15T15:17:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8df37fe640 1 --2- 172.21.15.46:0/1047477933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df40d2360 0x7f8df40d2b60 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8df40d1b00 tx=0x7f8de802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9104228b09093d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8df2ffd640 1 -- 172.21.15.46:0/1047477933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8035020 con 0x7f8df40d2360 2024-04-15T15:17:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8df2ffd640 1 -- 172.21.15.46:0/1047477933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8031830 con 0x7f8df40d2360 2024-04-15T15:17:47.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8df2ffd640 1 -- 172.21.15.46:0/1047477933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8031c00 con 0x7f8df40d2360 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 -- 172.21.15.46:0/1047477933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df40d2360 msgr2=0x7f8df40d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 --2- 172.21.15.46:0/1047477933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df40d2360 0x7f8df40d2b60 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f8df40d1b00 tx=0x7f8de802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 -- 172.21.15.46:0/1047477933 shutdown_connections 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 --2- 172.21.15.46:0/1047477933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df40d2360 0x7f8df40d2b60 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 --2- 172.21.15.46:0/1047477933 >> v2:172.21.15.78:3300/0 conn(0x7f8df415eae0 0x7f8df40d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 -- 172.21.15.46:0/1047477933 >> 172.21.15.46:0/1047477933 conn(0x7f8df408bf60 msgr2=0x7f8df40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.535+0000 7f8dfa277640 1 -- 172.21.15.46:0/1047477933 shutdown_connections 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.536+0000 7f8dfa277640 1 -- 172.21.15.46:0/1047477933 wait complete. 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.537+0000 7f8dfa277640 1 Processor -- start 2024-04-15T15:17:47.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.537+0000 7f8dfa277640 1 -- start start 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8df40d2360 0x7f8df40cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 0x7f8df414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8df415eae0 0x7f8df414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8df40d1d40 con 0x7f8df415e700 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f8df40b4450 con 0x7f8df40d2360 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f8df40d1ec0 con 0x7f8df415eae0 2024-04-15T15:17:47.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 0x7f8df414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 0x7f8df414d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41024/0 (socket says 172.21.15.46:41024) 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 -- 172.21.15.46:0/3270661613 learned_addr learned my addr 172.21.15.46:0/3270661613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df3fff640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8df40d2360 0x7f8df40cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df8814640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8df415eae0 0x7f8df414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 -- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8df415eae0 msgr2=0x7f8df414deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8df415eae0 0x7f8df414deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 -- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8df40d2360 msgr2=0x7f8df40cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8df40d2360 0x7f8df40cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 -- 172.21.15.46:0/3270661613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df4160f10 con 0x7f8df415e700 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df3fff640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8df40d2360 0x7f8df40cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df37fe640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 0x7f8df414d970 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8035040 con 0x7f8df415e700 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8004230 con 0x7f8df415e700 2024-04-15T15:17:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df41611f0 con 0x7f8df415e700 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de803a700 con 0x7f8df415e700 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.538+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df41617b0 con 0x7f8df415e700 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.540+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df414ba50 con 0x7f8df415e700 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.540+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8de8038020 con 0x7f8df415e700 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.540+0000 7f8df0ff9640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8de006c9b0 0x7f8de006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.540+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f8de80adbc0 con 0x7f8df415e700 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.541+0000 7f8df3fff640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8de006c9b0 0x7f8de006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:47.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.541+0000 7f8df3fff640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8de006c9b0 0x7f8de006ee70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8de40016f0 tx=0x7f8de4009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.545+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8de807bdb0 con 0x7f8df415e700 2024-04-15T15:17:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:47.768+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8df40689e0 con 0x7f8df415e700 2024-04-15T15:17:48.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.413+0000 7f8df0ff9640 1 -- 172.21.15.46:0/3270661613 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f8de807b6b0 con 0x7f8df415e700 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.415+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8de006c9b0 msgr2=0x7f8de006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.415+0000 7f8dfa277640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8de006c9b0 0x7f8de006ee70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8de40016f0 tx=0x7f8de4009040 comp rx=0 tx=0).stop 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.415+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 msgr2=0x7f8df414d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.415+0000 7f8dfa277640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 0x7f8df414d970 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8de8000c00 tx=0x7f8de80046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 shutdown_connections 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f8de006c9b0 0x7f8de006ee70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f8df415eae0 0x7f8df414deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8df415e700 0x7f8df414d970 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 --2- 172.21.15.46:0/3270661613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8df40d2360 0x7f8df40cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 >> 172.21.15.46:0/3270661613 conn(0x7f8df408bf60 msgr2=0x7f8df40cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:48.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 shutdown_connections 2024-04-15T15:17:48.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.416+0000 7f8dfa277640 1 -- 172.21.15.46:0/3270661613 wait complete. 2024-04-15T15:17:48.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:17:48.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.587+0000 7fb435c74640 1 Processor -- start 2024-04-15T15:17:48.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.587+0000 7fb435c74640 1 -- start start 2024-04-15T15:17:48.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb435c74640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb43015eb70 0x7fb4300b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb435c74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb43009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb435c74640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4301626e0 0x7fb430162b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb435c74640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb430163070 con 0x7fb4301626e0 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb435c74640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb4300d0d90 con 0x7fb43015eb70 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb435c74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb4300b1a20 con 0x7fb4300b1520 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4301626e0 0x7fb430162b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb4301626e0 0x7fb430162b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41038/0 (socket says 172.21.15.46:41038) 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42ffff640 1 -- 172.21.15.46:0/588694342 learned_addr learned my addr 172.21.15.46:0/588694342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42f7fe640 1 --2- 172.21.15.46:0/588694342 >> v2:172.21.15.78:3300/0 conn(0x7fb43015eb70 0x7fb4300b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42effd640 1 --2- 172.21.15.46:0/588694342 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb43009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42ffff640 1 -- 172.21.15.46:0/588694342 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 msgr2=0x7fb43009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.588+0000 7fb42ffff640 1 --2- 172.21.15.46:0/588694342 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb43009b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42ffff640 1 -- 172.21.15.46:0/588694342 >> v2:172.21.15.78:3300/0 conn(0x7fb43015eb70 msgr2=0x7fb4300b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42ffff640 1 --2- 172.21.15.46:0/588694342 >> v2:172.21.15.78:3300/0 conn(0x7fb43015eb70 0x7fb4300b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42ffff640 1 -- 172.21.15.46:0/588694342 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4300d0fe0 con 0x7fb4301626e0 2024-04-15T15:17:48.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42f7fe640 1 --2- 172.21.15.46:0/588694342 >> v2:172.21.15.78:3300/0 conn(0x7fb43015eb70 0x7fb4300b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42ffff640 1 --2- 172.21.15.46:0/588694342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4301626e0 0x7fb430162b30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb430073990 tx=0x7fb42802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c9bed4b6fa990b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42e7fc640 1 -- 172.21.15.46:0/588694342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42802c9e0 con 0x7fb4301626e0 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42e7fc640 1 -- 172.21.15.46:0/588694342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb42802cb80 con 0x7fb4301626e0 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.589+0000 7fb42e7fc640 1 -- 172.21.15.46:0/588694342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4280343f0 con 0x7fb4301626e0 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 -- 172.21.15.46:0/588694342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4301626e0 msgr2=0x7fb430162b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 --2- 172.21.15.46:0/588694342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4301626e0 0x7fb430162b30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb430073990 tx=0x7fb42802be40 comp rx=0 tx=0).stop 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 -- 172.21.15.46:0/588694342 shutdown_connections 2024-04-15T15:17:48.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 --2- 172.21.15.46:0/588694342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4301626e0 0x7fb430162b30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 --2- 172.21.15.46:0/588694342 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb43009b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 --2- 172.21.15.46:0/588694342 >> v2:172.21.15.78:3300/0 conn(0x7fb43015eb70 0x7fb4300b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 -- 172.21.15.46:0/588694342 >> 172.21.15.46:0/588694342 conn(0x7fb4300b3f70 msgr2=0x7fb4300b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 -- 172.21.15.46:0/588694342 shutdown_connections 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.590+0000 7fb435c74640 1 -- 172.21.15.46:0/588694342 wait complete. 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.591+0000 7fb435c74640 1 Processor -- start 2024-04-15T15:17:48.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 -- start start 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb430161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 0x7fb43014f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4301626e0 0x7fb4300c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb43009bd00 con 0x7fb43015eb70 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb4301634a0 con 0x7fb4301626e0 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb435c74640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb43006d160 con 0x7fb4300b1520 2024-04-15T15:17:48.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb42effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 0x7fb43014f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb42effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 0x7fb43014f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41042/0 (socket says 172.21.15.46:41042) 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb42effd640 1 -- 172.21.15.46:0/918416798 learned_addr learned my addr 172.21.15.46:0/918416798 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb42ffff640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4301626e0 0x7fb4300c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.592+0000 7fb42f7fe640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb430161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb42effd640 1 -- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 msgr2=0x7fb430161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb42effd640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb430161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb42effd640 1 -- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4301626e0 msgr2=0x7fb4300c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb42effd640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4301626e0 0x7fb4300c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb42effd640 1 -- 172.21.15.46:0/918416798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4300c9a90 con 0x7fb43015eb70 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb42effd640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 0x7fb43014f7c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb42400d800 tx=0x7fb42400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424018020 con 0x7fb43015eb70 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4240096a0 con 0x7fb43015eb70 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb430162450 con 0x7fb43015eb70 2024-04-15T15:17:48.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424009a70 con 0x7fb43015eb70 2024-04-15T15:17:48.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.593+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb430160680 con 0x7fb43015eb70 2024-04-15T15:17:48.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.594+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb430068250 con 0x7fb43015eb70 2024-04-15T15:17:48.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.595+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb42401b020 con 0x7fb43015eb70 2024-04-15T15:17:48.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.595+0000 7fb417fff640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb41c06c980 0x7fb41c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.595+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7fb42408be30 con 0x7fb43015eb70 2024-04-15T15:17:48.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.595+0000 7fb42f7fe640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb41c06c980 0x7fb41c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.596+0000 7fb42f7fe640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb41c06c980 0x7fb41c06ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb420006fd0 tx=0x7fb420008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.600+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb430068250 con 0x7fb43015eb70 2024-04-15T15:17:48.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:48.784+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb4300c5db0 con 0x7fb43015eb70 2024-04-15T15:17:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.414+0000 7fb417fff640 1 -- 172.21.15.46:0/918416798 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) ==== 181+0+0 (secure 0 0 0) 0x7fb4300c5db0 con 0x7fb43015eb70 2024-04-15T15:17:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:17:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb41c06c980 msgr2=0x7fb41c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb41c06c980 0x7fb41c06ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb420006fd0 tx=0x7fb420008040 comp rx=0 tx=0).stop 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 msgr2=0x7fb43014f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 0x7fb43014f7c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb42400d800 tx=0x7fb42400dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 shutdown_connections 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb41c06c980 0x7fb41c06ee40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb4301626e0 0x7fb4300c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb43015eb70 0x7fb43014f7c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 --2- 172.21.15.46:0/918416798 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb4300b1520 0x7fb430161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.417+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 >> 172.21.15.46:0/918416798 conn(0x7fb4300b3f70 msgr2=0x7fb43015a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.418+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 shutdown_connections 2024-04-15T15:17:49.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.418+0000 7fb435c74640 1 -- 172.21.15.46:0/918416798 wait complete. 2024-04-15T15:17:49.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:17:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.542+0000 7fb2f6cd9640 1 Processor -- start 2024-04-15T15:17:49.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.542+0000 7fb2f6cd9640 1 -- start start 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f6cd9640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb2f00d09b0 0x7fb2f00d0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f6cd9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2f00ce060 0x7fb2f009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f6cd9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f00d12d0 con 0x7fb2f009b920 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f6cd9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f00cdbd0 con 0x7fb2f00ce060 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f6cd9640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f00cd2a0 con 0x7fb2f00d09b0 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2effff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2f00ce060 0x7fb2f009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f5276640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb2f009b920 0x7fb2f00b1d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52108/0 (socket says 172.21.15.46:52108) 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.543+0000 7fb2f5276640 1 -- 172.21.15.46:0/478761932 learned_addr learned my addr 172.21.15.46:0/478761932 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.544+0000 7fb2f4a75640 1 --2- 172.21.15.46:0/478761932 >> v2:172.21.15.78:3300/0 conn(0x7fb2f00d09b0 0x7fb2f00d0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.544+0000 7fb2effff640 1 -- 172.21.15.46:0/478761932 >> v2:172.21.15.78:3300/0 conn(0x7fb2f00d09b0 msgr2=0x7fb2f00d0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:49.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.544+0000 7fb2effff640 1 --2- 172.21.15.46:0/478761932 >> v2:172.21.15.78:3300/0 conn(0x7fb2f00d09b0 0x7fb2f00d0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.544+0000 7fb2effff640 1 -- 172.21.15.46:0/478761932 >> v1:172.21.15.46:6789/0 conn(0x7fb2f009b920 legacy=0x7fb2f00b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.544+0000 7fb2ef7fe640 1 -- 172.21.15.46:0/478761932 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1759924398 0 0) 0x7fb2f00d12d0 con 0x7fb2f009b920 2024-04-15T15:17:49.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.544+0000 7fb2effff640 1 -- 172.21.15.46:0/478761932 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f00cd580 con 0x7fb2f00ce060 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2effff640 1 --2- 172.21.15.46:0/478761932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00ce060 0x7fb2f009b3e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0073980 tx=0x7fb2e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0f9864637fd3b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2ef7fe640 1 -- 172.21.15.46:0/478761932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4035020 con 0x7fb2f00ce060 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2ef7fe640 1 -- 172.21.15.46:0/478761932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e4031830 con 0x7fb2f00ce060 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2ef7fe640 1 -- 172.21.15.46:0/478761932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4031c00 con 0x7fb2f00ce060 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/478761932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00ce060 msgr2=0x7fb2f009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/478761932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00ce060 0x7fb2f009b3e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0073980 tx=0x7fb2e402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/478761932 shutdown_connections 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/478761932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00ce060 0x7fb2f009b3e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/478761932 >> v2:172.21.15.78:3300/0 conn(0x7fb2f00d09b0 0x7fb2f00d0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/478761932 >> 172.21.15.46:0/478761932 conn(0x7fb2f00b3ec0 msgr2=0x7fb2f00b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/478761932 shutdown_connections 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.545+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/478761932 wait complete. 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.546+0000 7fb2f6cd9640 1 Processor -- start 2024-04-15T15:17:49.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 -- start start 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2f009b920 0x7fb2f01f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2f00ce060 0x7fb2f00cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 0x7fb2f00b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f00cd060 con 0x7fb2f00d09b0 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f00ccee0 con 0x7fb2f009b920 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f6cd9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f00d0110 con 0x7fb2f00ce060 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.547+0000 7fb2f5276640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 0x7fb2f00b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 0x7fb2f00b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41800/0 (socket says 172.21.15.46:41800) 2024-04-15T15:17:49.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 -- 172.21.15.46:0/1276334002 learned_addr learned my addr 172.21.15.46:0/1276334002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2effff640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2f00ce060 0x7fb2f00cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f4a75640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2f009b920 0x7fb2f01f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 -- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2f00ce060 msgr2=0x7fb2f00cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2f00ce060 0x7fb2f00cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 -- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2f009b920 msgr2=0x7fb2f01f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2f009b920 0x7fb2f01f8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 -- 172.21.15.46:0/1276334002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f00b5000 con 0x7fb2f00d09b0 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2effff640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2f00ce060 0x7fb2f00cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f4a75640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2f009b920 0x7fb2f01f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f5276640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 0x7fb2f00b5ba0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000c3e0 tx=0x7fb2e000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00074e0 con 0x7fb2f00d09b0 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e00047e0 con 0x7fb2f00d09b0 2024-04-15T15:17:49.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f00b52e0 con 0x7fb2f00d09b0 2024-04-15T15:17:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0004bd0 con 0x7fb2f00d09b0 2024-04-15T15:17:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.548+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f015d970 con 0x7fb2f00d09b0 2024-04-15T15:17:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.549+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f00690f0 con 0x7fb2f00d09b0 2024-04-15T15:17:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.550+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb2e0011070 con 0x7fb2f00d09b0 2024-04-15T15:17:49.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.550+0000 7fb2ed7fa640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb2dc06c980 0x7fb2dc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.550+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5864+0+0 (secure 0 0 0) 0x7fb2e008b790 con 0x7fb2f00d09b0 2024-04-15T15:17:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.551+0000 7fb2f4a75640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb2dc06c980 0x7fb2dc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.551+0000 7fb2f4a75640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb2dc06c980 0x7fb2dc06ee40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8007330 tx=0x7fb2e800b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:49.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.555+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb2e0059b30 con 0x7fb2f00d09b0 2024-04-15T15:17:49.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:49.744+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb2f00b4b30 con 0x7fb2f00d09b0 2024-04-15T15:17:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.416+0000 7fb2ed7fa640 1 -- 172.21.15.46:0/1276334002 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) ==== 169+0+0 (secure 0 0 0) 0x7fb2e0059430 con 0x7fb2f00d09b0 2024-04-15T15:17:50.418 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-04-15T15:17:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb2dc06c980 msgr2=0x7fb2dc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb2dc06c980 0x7fb2dc06ee40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8007330 tx=0x7fb2e800b040 comp rx=0 tx=0).stop 2024-04-15T15:17:50.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 msgr2=0x7fb2f00b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 0x7fb2f00b5ba0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000c3e0 tx=0x7fb2e000c8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 shutdown_connections 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb2dc06c980 0x7fb2dc06ee40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2f00d09b0 0x7fb2f00b5ba0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb2f00ce060 0x7fb2f00cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 --2- 172.21.15.46:0/1276334002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb2f009b920 0x7fb2f01f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 >> 172.21.15.46:0/1276334002 conn(0x7fb2f00b3ec0 msgr2=0x7fb2f00b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 shutdown_connections 2024-04-15T15:17:50.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.419+0000 7fb2f6cd9640 1 -- 172.21.15.46:0/1276334002 wait complete. 2024-04-15T15:17:50.435 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-15T15:17:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.543+0000 7ff813ef2640 1 Processor -- start 2024-04-15T15:17:50.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.544+0000 7ff813ef2640 1 -- start start 2024-04-15T15:17:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.544+0000 7ff813ef2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff80c0b1620 0x7ff80c0b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.544+0000 7ff813ef2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff80c0af920 con 0x7ff80c15ecb0 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.544+0000 7ff813ef2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff80c0afc00 con 0x7ff80c0af340 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.544+0000 7ff813ef2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff80c15f090 con 0x7ff80c0b1620 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.544+0000 7ff81248f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff80c0b1620 0x7ff80c0b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.545+0000 7ff811c8e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff80c15ecb0 0x7ff80c0b0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59276/0 (socket says 172.21.15.46:59276) 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.545+0000 7ff811c8e640 1 -- 172.21.15.46:0/1840377346 learned_addr learned my addr 172.21.15.46:0/1840377346 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.545+0000 7ff81248f640 1 -- 172.21.15.46:0/1840377346 >> v1:172.21.15.78:6789/0 conn(0x7ff80c0af340 legacy=0x7ff80c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.545+0000 7ff81248f640 1 -- 172.21.15.46:0/1840377346 >> v1:172.21.15.46:6789/0 conn(0x7ff80c15ecb0 legacy=0x7ff80c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.545+0000 7ff810c8c640 1 -- 172.21.15.46:0/1840377346 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3149975252 0 0) 0x7ff80c0af920 con 0x7ff80c15ecb0 2024-04-15T15:17:50.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.545+0000 7ff81248f640 1 -- 172.21.15.46:0/1840377346 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c0afee0 con 0x7ff80c0b1620 2024-04-15T15:17:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff81248f640 1 --2- 172.21.15.46:0/1840377346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c0b1ab0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff8040097a0 tx=0x7ff80402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793a2c96cfa3ed3e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff810c8c640 1 -- 172.21.15.46:0/1840377346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804034070 con 0x7ff80c0b1620 2024-04-15T15:17:50.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff810c8c640 1 -- 172.21.15.46:0/1840377346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff804030610 con 0x7ff80c0b1620 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff810c8c640 1 -- 172.21.15.46:0/1840377346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8040309e0 con 0x7ff80c0b1620 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 -- 172.21.15.46:0/1840377346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 msgr2=0x7ff80c0b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 --2- 172.21.15.46:0/1840377346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c0b1ab0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff8040097a0 tx=0x7ff80402b960 comp rx=0 tx=0).stop 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 -- 172.21.15.46:0/1840377346 shutdown_connections 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 --2- 172.21.15.46:0/1840377346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c0b1ab0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 -- 172.21.15.46:0/1840377346 >> 172.21.15.46:0/1840377346 conn(0x7ff80c0b6870 msgr2=0x7ff80c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 -- 172.21.15.46:0/1840377346 shutdown_connections 2024-04-15T15:17:50.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.546+0000 7ff813ef2640 1 -- 172.21.15.46:0/1840377346 wait complete. 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 Processor -- start 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 -- start start 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff80c0af340 0x7ff80c0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c09b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff80c15ecb0 0x7ff80c09d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c0ce7a0 con 0x7ff80c0b1620 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c0ce620 con 0x7ff80c0af340 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff813ef2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c0ce920 con 0x7ff80c15ecb0 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff81148d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c09b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff81148d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c09b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41816/0 (socket says 172.21.15.46:41816) 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.548+0000 7ff81148d640 1 -- 172.21.15.46:0/2615536696 learned_addr learned my addr 172.21.15.46:0/2615536696 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff811c8e640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff80c0af340 0x7ff80c0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81248f640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff80c15ecb0 0x7ff80c09d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81148d640 1 -- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff80c15ecb0 msgr2=0x7ff80c09d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81148d640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff80c15ecb0 0x7ff80c09d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81148d640 1 -- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff80c0af340 msgr2=0x7ff80c0ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81148d640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff80c0af340 0x7ff80c0ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81148d640 1 -- 172.21.15.46:0/2615536696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c14dc60 con 0x7ff80c0b1620 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff811c8e640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff80c0af340 0x7ff80c0ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81248f640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff80c15ecb0 0x7ff80c09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff81148d640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c09b4b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff80000c430 tx=0x7ff80000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800011790 con 0x7ff80c0b1620 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff800017070 con 0x7ff80c0b1620 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c14df40 con 0x7ff80c0b1620 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff800011bb0 con 0x7ff80c0b1620 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.549+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c14e500 con 0x7ff80c0b1620 2024-04-15T15:17:50.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.550+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80c0689e0 con 0x7ff80c0b1620 2024-04-15T15:17:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.551+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff80001c460 con 0x7ff80c0b1620 2024-04-15T15:17:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.551+0000 7ff80a7fc640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff7f806c980 0x7ff7f806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.551+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7ff80008b980 con 0x7ff80c0b1620 2024-04-15T15:17:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.551+0000 7ff811c8e640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff7f806c980 0x7ff7f806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.552+0000 7ff811c8e640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff7f806c980 0x7ff7f806ee40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc007fd0 tx=0x7ff7fc009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:50.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.556+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff800059ee0 con 0x7ff80c0b1620 2024-04-15T15:17:50.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.739+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7ff80c06c7c0 con 0x7ff80c0b1620 2024-04-15T15:17:50.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.742+0000 7ff80a7fc640 1 -- 172.21.15.46:0/2615536696 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7ff8000597e0 con 0x7ff80c0b1620 2024-04-15T15:17:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff7f806c980 msgr2=0x7ff7f806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff7f806c980 0x7ff7f806ee40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc007fd0 tx=0x7ff7fc009040 comp rx=0 tx=0).stop 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 msgr2=0x7ff80c09b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c09b4b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7ff80000c430 tx=0x7ff80000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 shutdown_connections 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff7f806c980 0x7ff7f806ee40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff80c15ecb0 0x7ff80c09d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff80c0b1620 0x7ff80c09b4b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 --2- 172.21.15.46:0/2615536696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff80c0af340 0x7ff80c0ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 >> 172.21.15.46:0/2615536696 conn(0x7ff80c0b6870 msgr2=0x7ff80c15df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 shutdown_connections 2024-04-15T15:17:50.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.744+0000 7ff813ef2640 1 -- 172.21.15.46:0/2615536696 wait complete. 2024-04-15T15:17:50.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-15T15:17:50.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.867+0000 7feb40d75640 1 Processor -- start 2024-04-15T15:17:50.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.867+0000 7feb40d75640 1 -- start start 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.868+0000 7feb40d75640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7feb3c15e700 0x7feb3c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.868+0000 7feb40d75640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3c0b47a0 con 0x7feb3c0d2360 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.868+0000 7feb40d75640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3c0b4a80 con 0x7feb3c0d3270 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.868+0000 7feb40d75640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7feb3c0b53b0 con 0x7feb3c15e700 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.868+0000 7feb39d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feb3c0d2360 0x7feb3c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59286/0 (socket says 172.21.15.46:59286) 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.868+0000 7feb39d9b640 1 -- 172.21.15.46:0/1370966260 learned_addr learned my addr 172.21.15.46:0/1370966260 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 --2- 172.21.15.46:0/1370966260 >> v2:172.21.15.78:3300/0 conn(0x7feb3c15e700 0x7feb3c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1514034444 0 0) 0x7feb3c0b47a0 con 0x7feb3c0d2360 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb28003520 con 0x7feb3c0d2360 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1970077832 0 0) 0x7feb28003520 con 0x7feb3c0d2360 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 >> v2:172.21.15.78:3300/0 conn(0x7feb3c15e700 msgr2=0x7feb3c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 --2- 172.21.15.46:0/1370966260 >> v2:172.21.15.78:3300/0 conn(0x7feb3c15e700 0x7feb3c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 >> v1:172.21.15.78:6789/0 conn(0x7feb3c0d3270 legacy=0x7feb3c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c0b4d60 con 0x7feb3c0d2360 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7feb24001670 con 0x7feb3c0d2360 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 >> v1:172.21.15.46:6789/0 conn(0x7feb3c0d2360 legacy=0x7feb3c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280039b0 0x7feb28003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb28004800 0x7feb28004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb280051c0 0x7feb2800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c0b53b0 con 0x7feb280039b0 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feb2800c090 con 0x7feb28004800 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7feb2800c290 con 0x7feb280051c0 2024-04-15T15:17:50.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280039b0 0x7feb28003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3ad9d640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb280051c0 0x7feb2800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb39d9b640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb28004800 0x7feb28004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 -- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb280051c0 msgr2=0x7feb2800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb280051c0 0x7feb2800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 -- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb28004800 msgr2=0x7feb28004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb28004800 0x7feb28004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3a59c640 1 -- 172.21.15.46:0/1370966260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb28003520 con 0x7feb280039b0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.869+0000 7feb3ad9d640 1 -- 172.21.15.46:0/1370966260 reap_dead start 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.870+0000 7feb3a59c640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280039b0 0x7feb28003df0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7feb34009ab0 tx=0x7feb28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2621a6643e3ed2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.870+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34009680 con 0x7feb280039b0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.870+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb34005990 con 0x7feb280039b0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.870+0000 7feb3959a640 1 -- 172.21.15.46:0/1370966260 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb34005d80 con 0x7feb280039b0 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 -- 172.21.15.46:0/1370966260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280039b0 msgr2=0x7feb28003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280039b0 0x7feb28003df0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7feb34009ab0 tx=0x7feb28005830 comp rx=0 tx=0).stop 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 -- 172.21.15.46:0/1370966260 shutdown_connections 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 --2- 172.21.15.46:0/1370966260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb280039b0 0x7feb28003df0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 -- 172.21.15.46:0/1370966260 >> 172.21.15.46:0/1370966260 conn(0x7feb3c08bf60 msgr2=0x7feb3c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 -- 172.21.15.46:0/1370966260 shutdown_connections 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.871+0000 7feb40d75640 1 -- 172.21.15.46:0/1370966260 wait complete. 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.872+0000 7feb40d75640 1 Processor -- start 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.872+0000 7feb40d75640 1 -- start start 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb40d75640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280039b0 0x7feb3c14e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb40d75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 0x7feb3c0b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb40d75640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb3c0b7600 0x7feb3c0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb40d75640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c0d1d40 con 0x7feb3c0b7180 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb40d75640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7feb3c0b4450 con 0x7feb280039b0 2024-04-15T15:17:50.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb40d75640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7feb3c0d1ec0 con 0x7feb3c0b7600 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 0x7feb3c0b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 0x7feb3c0b1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41836/0 (socket says 172.21.15.46:41836) 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 -- 172.21.15.46:0/2350655041 learned_addr learned my addr 172.21.15.46:0/2350655041 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb3ad9d640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb3c0b7600 0x7feb3c0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb3a59c640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280039b0 0x7feb3c14e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 -- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb3c0b7600 msgr2=0x7feb3c0b1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb3c0b7600 0x7feb3c0b1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 -- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280039b0 msgr2=0x7feb3c14e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280039b0 0x7feb3c14e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb39d9b640 1 -- 172.21.15.46:0/2350655041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3c14ef00 con 0x7feb3c0b7180 2024-04-15T15:17:50.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.873+0000 7feb3a59c640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280039b0 0x7feb3c14e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.874+0000 7feb39d9b640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 0x7feb3c0b1590 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feb2400e870 tx=0x7feb2400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.874+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2400c870 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.874+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb24018070 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.874+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb3c0b9df0 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.874+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2400cc70 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.874+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3c0cbd10 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.875+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb3c0b4700 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.875+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7feb2401d460 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.876+0000 7feb32ffd640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feb1c06c980 0x7feb1c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.876+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7feb2408b7e0 con 0x7feb3c0b7180 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.876+0000 7feb3a59c640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feb1c06c980 0x7feb1c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:50.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.876+0000 7feb3a59c640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feb1c06c980 0x7feb1c06ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feb34009ab0 tx=0x7feb34030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:50.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:50.880+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb3c0b4700 con 0x7feb3c0b7180 2024-04-15T15:17:51.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.065+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7feb3c0689e0 con 0x7feb3c0b7180 2024-04-15T15:17:51.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.068+0000 7feb32ffd640 1 -- 172.21.15.46:0/2350655041 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7feb3c14ef00 con 0x7feb3c0b7180 2024-04-15T15:17:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.070+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feb1c06c980 msgr2=0x7feb1c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.070+0000 7feb40d75640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feb1c06c980 0x7feb1c06ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feb34009ab0 tx=0x7feb34030000 comp rx=0 tx=0).stop 2024-04-15T15:17:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 msgr2=0x7feb3c0b1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 0x7feb3c0b1590 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7feb2400e870 tx=0x7feb2400ed30 comp rx=0 tx=0).stop 2024-04-15T15:17:51.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 shutdown_connections 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7feb1c06c980 0x7feb1c06ee40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7feb3c0b7600 0x7feb3c0b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb3c0b7180 0x7feb3c0b1590 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 --2- 172.21.15.46:0/2350655041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7feb280039b0 0x7feb3c14e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 >> 172.21.15.46:0/2350655041 conn(0x7feb3c08bf60 msgr2=0x7feb3c0c81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 shutdown_connections 2024-04-15T15:17:51.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.071+0000 7feb40d75640 1 -- 172.21.15.46:0/2350655041 wait complete. 2024-04-15T15:17:51.084 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_cache_quiesce_splitauth 2024-04-15T15:17:51.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.189+0000 7f3eaca4a640 1 Processor -- start 2024-04-15T15:17:51.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.189+0000 7f3eaca4a640 1 -- start start 2024-04-15T15:17:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3eaca4a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3ea809ac40 0x7f3ea80d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3eaca4a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ea80b49d0 0x7f3ea80ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3eaca4a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea80d1210 con 0x7f3ea80ce570 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3eaca4a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea80b1b70 con 0x7f3ea80b49d0 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3eaca4a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea80d14b0 con 0x7f3ea809ac40 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3ea5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ea80b49d0 0x7f3ea80ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3ea5d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3ea80b49d0 0x7f3ea80ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41844/0 (socket says 172.21.15.46:41844) 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.190+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/2135556657 learned_addr learned my addr 172.21.15.46:0/2135556657 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.191+0000 7f3ea659c640 1 --2- 172.21.15.46:0/2135556657 >> v2:172.21.15.78:3300/0 conn(0x7f3ea809ac40 0x7f3ea80d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.191+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/2135556657 >> v2:172.21.15.78:3300/0 conn(0x7f3ea809ac40 msgr2=0x7f3ea80d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.191+0000 7f3ea5d9b640 1 --2- 172.21.15.46:0/2135556657 >> v2:172.21.15.78:3300/0 conn(0x7f3ea809ac40 0x7f3ea80d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.191+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/2135556657 >> v1:172.21.15.78:6789/0 conn(0x7f3ea80ce570 legacy=0x7f3ea80b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:51.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.191+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/2135556657 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea80cd550 con 0x7f3ea80b49d0 2024-04-15T15:17:51.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3ea5d9b640 1 --2- 172.21.15.46:0/2135556657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80ce030 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3ea81613d0 tx=0x7f3e9002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc41434c90514dad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3ea559a640 1 -- 172.21.15.46:0/2135556657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90034070 con 0x7f3ea80b49d0 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3ea559a640 1 -- 172.21.15.46:0/2135556657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e90030860 con 0x7f3ea80b49d0 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3ea559a640 1 -- 172.21.15.46:0/2135556657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90030c30 con 0x7f3ea80b49d0 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 -- 172.21.15.46:0/2135556657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 msgr2=0x7f3ea80ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/2135556657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80ce030 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3ea81613d0 tx=0x7f3e9002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 -- 172.21.15.46:0/2135556657 shutdown_connections 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/2135556657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80ce030 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/2135556657 >> v2:172.21.15.78:3300/0 conn(0x7f3ea809ac40 0x7f3ea80d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 -- 172.21.15.46:0/2135556657 >> 172.21.15.46:0/2135556657 conn(0x7f3ea80b3eb0 msgr2=0x7f3ea80b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 -- 172.21.15.46:0/2135556657 shutdown_connections 2024-04-15T15:17:51.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.192+0000 7f3eaca4a640 1 -- 172.21.15.46:0/2135556657 wait complete. 2024-04-15T15:17:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 Processor -- start 2024-04-15T15:17:51.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 -- start start 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ea809ac40 0x7f3ea80c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ea80ce570 0x7f3ea81566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea80b2430 con 0x7f3ea80b49d0 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.194+0000 7f3eaca4a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea80cd1e0 con 0x7f3ea80ce570 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3eaca4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea80b25b0 con 0x7f3ea809ac40 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80c6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/691875936 learned_addr learned my addr 172.21.15.46:0/691875936 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea659c640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ea809ac40 0x7f3ea80c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea6d9d640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ea80ce570 0x7f3ea81566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ea809ac40 msgr2=0x7f3ea80c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ea809ac40 0x7f3ea80c9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ea80ce570 msgr2=0x7f3ea81566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ea80ce570 0x7f3ea81566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 -- 172.21.15.46:0/691875936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea80c99d0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3ea5d9b640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80c6170 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3e9002b9e0 tx=0x7f3e90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90048070 con 0x7f3ea80b49d0 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e90030660 con 0x7f3ea80b49d0 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e90034040 con 0x7f3ea80b49d0 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea80c66b0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.195+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea80b63b0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.196+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea80689e0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.197+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3e90037070 con 0x7f3ea80b49d0 2024-04-15T15:17:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.197+0000 7f3e9effd640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3e8806c980 0x7f3e8806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.197+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f3e900adc60 con 0x7f3ea80b49d0 2024-04-15T15:17:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.198+0000 7f3ea659c640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3e8806c980 0x7f3e8806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.198+0000 7f3ea659c640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3e8806c980 0x7f3e8806ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0006fd0 tx=0x7f3ea0008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:51.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.202+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e9007c0d0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.386+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"} v 0) -- 0x7f3ea80d1db0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.389+0000 7f3e9effd640 1 -- 172.21.15.46:0/691875936 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"}]=0 v15) ==== 110+0+0 (secure 0 0 0) 0x7f3e9007b9d0 con 0x7f3ea80b49d0 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3e8806c980 msgr2=0x7f3e8806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3e8806c980 0x7f3e8806ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0006fd0 tx=0x7f3ea0008040 comp rx=0 tx=0).stop 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 msgr2=0x7f3ea80c6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80c6170 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3e9002b9e0 tx=0x7f3e90004730 comp rx=0 tx=0).stop 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 shutdown_connections 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3e8806c980 0x7f3e8806ee40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ea80ce570 0x7f3ea81566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ea80b49d0 0x7f3ea80c6170 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 --2- 172.21.15.46:0/691875936 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3ea809ac40 0x7f3ea80c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 >> 172.21.15.46:0/691875936 conn(0x7f3ea80b3eb0 msgr2=0x7f3ea80b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 shutdown_connections 2024-04-15T15:17:51.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.391+0000 7f3eaca4a640 1 -- 172.21.15.46:0/691875936 wait complete. 2024-04-15T15:17:51.404 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:17:51.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 Processor -- start 2024-04-15T15:17:51.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 -- start start 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff31809a9d0 0x7ff31809adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff31815ebb0 0x7ff31806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff31809b2f0 con 0x7ff31809b780 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff3180cd8c0 con 0x7ff31809a9d0 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.508+0000 7ff31e48f640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff3180ced60 con 0x7ff31815ebb0 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff31809a9d0 0x7ff31809adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff31809a9d0 0x7ff31809adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41862/0 (socket says 172.21.15.46:41862) 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 -- 172.21.15.46:0/3858752109 learned_addr learned my addr 172.21.15.46:0/3858752109 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 -- 172.21.15.46:0/3858752109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff31815ebb0 msgr2=0x7ff31806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff30ffff640 1 --2- 172.21.15.46:0/3858752109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff31815ebb0 0x7ff31806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 --2- 172.21.15.46:0/3858752109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff31815ebb0 0x7ff31806cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 -- 172.21.15.46:0/3858752109 >> v1:172.21.15.78:6789/0 conn(0x7ff31809b780 legacy=0x7ff3180b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:51.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.509+0000 7ff317fff640 1 -- 172.21.15.46:0/3858752109 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3180cccb0 con 0x7ff31809a9d0 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff317fff640 1 --2- 172.21.15.46:0/3858752109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff31809adb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff308009870 tx=0x7ff30802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f16ae19f7a7ffd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff3177fe640 1 -- 172.21.15.46:0/3858752109 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308035020 con 0x7ff31809a9d0 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff3177fe640 1 -- 172.21.15.46:0/3858752109 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff308031640 con 0x7ff31809a9d0 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff3177fe640 1 -- 172.21.15.46:0/3858752109 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30802d040 con 0x7ff31809a9d0 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff31e48f640 1 -- 172.21.15.46:0/3858752109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 msgr2=0x7ff31809adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3858752109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff31809adb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7ff308009870 tx=0x7ff30802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff31e48f640 1 -- 172.21.15.46:0/3858752109 shutdown_connections 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3858752109 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff31815ebb0 0x7ff31806cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3858752109 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff31809adb0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.510+0000 7ff31e48f640 1 -- 172.21.15.46:0/3858752109 >> 172.21.15.46:0/3858752109 conn(0x7ff3180b3fb0 msgr2=0x7ff3180b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:51.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.511+0000 7ff31e48f640 1 -- 172.21.15.46:0/3858752109 shutdown_connections 2024-04-15T15:17:51.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.511+0000 7ff31e48f640 1 -- 172.21.15.46:0/3858752109 wait complete. 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 Processor -- start 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 -- start start 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff3180c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff31809b780 0x7ff3181519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff3180b5da0 0x7ff3180cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff3180cd550 con 0x7ff31809a9d0 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff3180cd270 con 0x7ff31809b780 2024-04-15T15:17:51.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff31e48f640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff31806d240 con 0x7ff3180b5da0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.512+0000 7ff317fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff3180c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff3180c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41866/0 (socket says 172.21.15.46:41866) 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 -- 172.21.15.46:0/3273720955 learned_addr learned my addr 172.21.15.46:0/3273720955 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff31ca2c640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff3180b5da0 0x7ff3180cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 -- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff3180b5da0 msgr2=0x7ff3180cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff30ffff640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff31809b780 0x7ff3181519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff3180b5da0 0x7ff3180cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 -- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff31809b780 msgr2=0x7ff3181519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff31809b780 0x7ff3181519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 -- 172.21.15.46:0/3273720955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3180c9b70 con 0x7ff31809a9d0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff317fff640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff3180c9590 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff308000c00 tx=0x7ff3080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308040070 con 0x7ff31809a9d0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff308039bf0 con 0x7ff31809a9d0 2024-04-15T15:17:51.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3180c9e50 con 0x7ff31809a9d0 2024-04-15T15:17:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff308035040 con 0x7ff31809a9d0 2024-04-15T15:17:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.513+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3180ca410 con 0x7ff31809a9d0 2024-04-15T15:17:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.514+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3180690f0 con 0x7ff31809a9d0 2024-04-15T15:17:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.515+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff30804b060 con 0x7ff31809a9d0 2024-04-15T15:17:51.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.515+0000 7ff3157fa640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f806c980 0x7ff2f806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.515+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7ff3080ad5b0 con 0x7ff31809a9d0 2024-04-15T15:17:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.515+0000 7ff30ffff640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f806c980 0x7ff2f806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:51.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.516+0000 7ff30ffff640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f806c980 0x7ff2f806ee40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff31815b470 tx=0x7ff300008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:51.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.519+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff308031610 con 0x7ff31809a9d0 2024-04-15T15:17:51.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:51.775+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7ff31806ed00 con 0x7ff31809a9d0 2024-04-15T15:17:52.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.421+0000 7ff3157fa640 1 -- 172.21.15.46:0/3273720955 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v149) ==== 137+0+0 (secure 0 0 0) 0x7ff30807bb10 con 0x7ff31809a9d0 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f806c980 msgr2=0x7ff2f806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f806c980 0x7ff2f806ee40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff31815b470 tx=0x7ff300008040 comp rx=0 tx=0).stop 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 msgr2=0x7ff3180c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff3180c9590 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff308000c00 tx=0x7ff3080046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 shutdown_connections 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff2f806c980 0x7ff2f806ee40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff3180b5da0 0x7ff3180cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff31809b780 0x7ff3181519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 --2- 172.21.15.46:0/3273720955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff31809a9d0 0x7ff3180c9590 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:52.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 >> 172.21.15.46:0/3273720955 conn(0x7ff3180b3fb0 msgr2=0x7ff3180d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:52.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 shutdown_connections 2024-04-15T15:17:52.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:52.423+0000 7ff31e48f640 1 -- 172.21.15.46:0/3273720955 wait complete. 2024-04-15T15:17:52.439 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. ... FAIL 2024-04-15T15:17:52.439 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:52.439 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:52.440 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:Ran 1 test in 86.134s 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:17:52.441 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:17:52.442 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:17:52.442 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:17:52.442 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:17:52.442 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:17:52.442 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:17:52.442 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:52.442 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:52.693 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=55ccec76d8c44a38bd30e3295474c246 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:52.694 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-15T15:17:52.705 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-15T15:17:52.747 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-15T15:17:52.748 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:52.748 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:52.770 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15004 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:52.770 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-15T15:17:52.770 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:52.828 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15004 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:52.828 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:52.828 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:52.847 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056355,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16861 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:52.848 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-04-15T15:17:52.848 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:52.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056355,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16861 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497668k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:52.906 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2024-04-15T15:17:52.963 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:17:52.963 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:17:52.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:52.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:53.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:53.035 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:53.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:53.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2024-04-15T15:17:53.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:53.210 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-15T15:17:53.229 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:17:53.230 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:17:53.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:53.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:53.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:53.304 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:53.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:53.377 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-04-15T15:17:53.488 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-15T15:17:53.498 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:17:53.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:53.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.603+0000 7fdb637e2640 1 Processor -- start 2024-04-15T15:17:53.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.603+0000 7fdb637e2640 1 -- start start 2024-04-15T15:17:53.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb637e2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb5c0d0690 0x7fdb5c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb637e2640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdb5c0ae280 0x7fdb5c0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb637e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb5c0d0fb0 con 0x7fdb5c0b3160 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb637e2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5c15bda0 con 0x7fdb5c0d0690 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb637e2640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5c15cb50 con 0x7fdb5c0ae280 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb5c0d0690 0x7fdb5c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdb5c0d0690 0x7fdb5c0d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41880/0 (socket says 172.21.15.46:41880) 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 -- 172.21.15.46:0/2755983002 learned_addr learned my addr 172.21.15.46:0/2755983002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:53.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb61d7f640 1 --2- 172.21.15.46:0/2755983002 >> v2:172.21.15.78:3300/0 conn(0x7fdb5c0ae280 0x7fdb5c0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 -- 172.21.15.46:0/2755983002 >> v2:172.21.15.78:3300/0 conn(0x7fdb5c0ae280 msgr2=0x7fdb5c0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:53.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 --2- 172.21.15.46:0/2755983002 >> v2:172.21.15.78:3300/0 conn(0x7fdb5c0ae280 0x7fdb5c0ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 -- 172.21.15.46:0/2755983002 >> v1:172.21.15.78:6789/0 conn(0x7fdb5c0b3160 legacy=0x7fdb5c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.604+0000 7fdb6157e640 1 -- 172.21.15.46:0/2755983002 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c15bfd0 con 0x7fdb5c0d0690 2024-04-15T15:17:53.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.605+0000 7fdb6157e640 1 --2- 172.21.15.46:0/2755983002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c0d0a70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0b1df0 tx=0x7fdb5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54582e46012c8f59 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:53.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.605+0000 7fdb5bfff640 1 -- 172.21.15.46:0/2755983002 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54034070 con 0x7fdb5c0d0690 2024-04-15T15:17:53.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.605+0000 7fdb5bfff640 1 -- 172.21.15.46:0/2755983002 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb54030860 con 0x7fdb5c0d0690 2024-04-15T15:17:53.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.605+0000 7fdb5bfff640 1 -- 172.21.15.46:0/2755983002 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54030c30 con 0x7fdb5c0d0690 2024-04-15T15:17:53.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 -- 172.21.15.46:0/2755983002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0d0690 msgr2=0x7fdb5c0d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 --2- 172.21.15.46:0/2755983002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c0d0a70 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0b1df0 tx=0x7fdb5402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 -- 172.21.15.46:0/2755983002 shutdown_connections 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 --2- 172.21.15.46:0/2755983002 >> v2:172.21.15.78:3300/0 conn(0x7fdb5c0ae280 0x7fdb5c0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 --2- 172.21.15.46:0/2755983002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c0d0a70 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 -- 172.21.15.46:0/2755983002 >> 172.21.15.46:0/2755983002 conn(0x7fdb5c06be00 msgr2=0x7fdb5c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 -- 172.21.15.46:0/2755983002 shutdown_connections 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.606+0000 7fdb637e2640 1 -- 172.21.15.46:0/2755983002 wait complete. 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.607+0000 7fdb637e2640 1 Processor -- start 2024-04-15T15:17:53.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.607+0000 7fdb637e2640 1 -- start start 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb637e2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 0x7fdb5c0b2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb637e2640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdb5c0b3160 0x7fdb5c148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb637e2640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c1486c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb637e2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c15c870 con 0x7fdb5c0ae280 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb637e2640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c15c590 con 0x7fdb5c0d0690 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb637e2640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5c0b4fb0 con 0x7fdb5c0b3160 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 0x7fdb5c0b2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 0x7fdb5c0b2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41894/0 (socket says 172.21.15.46:41894) 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 -- 172.21.15.46:0/312249130 learned_addr learned my addr 172.21.15.46:0/312249130 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb60d7d640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdb5c0b3160 0x7fdb5c148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb61d7f640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c1486c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 -- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdb5c0b3160 msgr2=0x7fdb5c148180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdb5c0b3160 0x7fdb5c148180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 -- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb5c0d0690 msgr2=0x7fdb5c1486c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c1486c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb6157e640 1 -- 172.21.15.46:0/312249130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c1544b0 con 0x7fdb5c0ae280 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.608+0000 7fdb60d7d640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdb5c0b3160 0x7fdb5c148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.609+0000 7fdb6157e640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 0x7fdb5c0b2050 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fdb5402b9e0 tx=0x7fdb54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.609+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54048070 con 0x7fdb5c0ae280 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.609+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb54030660 con 0x7fdb5c0ae280 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.609+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5c154790 con 0x7fdb5c0ae280 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.609+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb54034040 con 0x7fdb5c0ae280 2024-04-15T15:17:53.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.609+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb5c154d50 con 0x7fdb5c0ae280 2024-04-15T15:17:53.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.610+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb5c0689e0 con 0x7fdb5c0ae280 2024-04-15T15:17:53.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.611+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fdb5404b070 con 0x7fdb5c0ae280 2024-04-15T15:17:53.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.611+0000 7fdb59ffb640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdb4806c980 0x7fdb4806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.611+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fdb540ada30 con 0x7fdb5c0ae280 2024-04-15T15:17:53.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.611+0000 7fdb60d7d640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdb4806c980 0x7fdb4806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.612+0000 7fdb60d7d640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdb4806c980 0x7fdb4806ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c153a20 tx=0x7fdb50009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:53.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.615+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb5407bf10 con 0x7fdb5c0ae280 2024-04-15T15:17:53.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.798+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb5c0739e0 con 0x7fdb4806c980 2024-04-15T15:17:53.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.799+0000 7fdb59ffb640 1 -- 172.21.15.46:0/312249130 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13479 (secure 0 0 0) 0x7fdb5c0739e0 con 0x7fdb4806c980 2024-04-15T15:17:53.801 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:53.801 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:53.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdb4806c980 msgr2=0x7fdb4806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdb4806c980 0x7fdb4806ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c153a20 tx=0x7fdb50009040 comp rx=0 tx=0).stop 2024-04-15T15:17:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 msgr2=0x7fdb5c0b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 0x7fdb5c0b2050 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fdb5402b9e0 tx=0x7fdb54004730 comp rx=0 tx=0).stop 2024-04-15T15:17:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 shutdown_connections 2024-04-15T15:17:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdb4806c980 0x7fdb4806ee40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb5c0d0690 0x7fdb5c1486c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdb5c0b3160 0x7fdb5c148180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 --2- 172.21.15.46:0/312249130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb5c0ae280 0x7fdb5c0b2050 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 >> 172.21.15.46:0/312249130 conn(0x7fdb5c06be00 msgr2=0x7fdb5c0cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 shutdown_connections 2024-04-15T15:17:53.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.801+0000 7fdb637e2640 1 -- 172.21.15.46:0/312249130 wait complete. 2024-04-15T15:17:53.813 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-15T15:17:52.386130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":238020,"kb_used_data":5196,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736700,"statfs":{"total":773094113280,"available":772850380800,"internally_reserved":0,"allocated":5320704,"data_stored":3002173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61070,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-128,"num_write":-92,"num_write_kb":-94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001392"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:49.420142+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:49.420142+0000","last_peered":"2024-04-15T15:17:49.420142+0000","last_clean":"2024-04-15T15:17:49.420142+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:49.420142+0000","last_undegraded":"2024-04-15T15:17:49.420142+0000","last_fullsized":"2024-04-15T15:17:49.420142+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542177,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542177,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27520,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1110016,"data_stored":816649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":816516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:53.814 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.917+0000 7f64dbeb1640 1 Processor -- start 2024-04-15T15:17:53.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.917+0000 7f64dbeb1640 1 -- start start 2024-04-15T15:17:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64dbeb1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d40701f0 0x7f64d4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64dbeb1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f64d415fb30 0x7f64d40d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64dbeb1640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64d4070720 con 0x7f64d409a9d0 2024-04-15T15:17:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64dbeb1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f64d409ba90 con 0x7f64d415fb30 2024-04-15T15:17:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64dbeb1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f64d40d08c0 con 0x7f64d40701f0 2024-04-15T15:17:53.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d9c4d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f64d409a9d0 0x7f64d409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59288/0 (socket says 172.21.15.46:59288) 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d9c4d640 1 -- 172.21.15.46:0/2592336708 learned_addr learned my addr 172.21.15.46:0/2592336708 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d944c640 1 --2- 172.21.15.46:0/2592336708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d40701f0 0x7f64d4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64da44e640 1 --2- 172.21.15.46:0/2592336708 >> v2:172.21.15.78:3300/0 conn(0x7f64d415fb30 0x7f64d40d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d8c4b640 1 -- 172.21.15.46:0/2592336708 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667132362 0 0) 0x7f64d4070720 con 0x7f64d409a9d0 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d8c4b640 1 -- 172.21.15.46:0/2592336708 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c0003520 con 0x7f64d409a9d0 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d944c640 1 -- 172.21.15.46:0/2592336708 >> v2:172.21.15.78:3300/0 conn(0x7f64d415fb30 msgr2=0x7f64d40d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d944c640 1 --2- 172.21.15.46:0/2592336708 >> v2:172.21.15.78:3300/0 conn(0x7f64d415fb30 0x7f64d40d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d944c640 1 -- 172.21.15.46:0/2592336708 >> v1:172.21.15.46:6789/0 conn(0x7f64d409a9d0 legacy=0x7f64d409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64d944c640 1 -- 172.21.15.46:0/2592336708 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d4070a90 con 0x7f64d40701f0 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.918+0000 7f64da44e640 1 --2- 172.21.15.46:0/2592336708 >> v2:172.21.15.78:3300/0 conn(0x7f64d415fb30 0x7f64d40d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.919+0000 7f64d944c640 1 --2- 172.21.15.46:0/2592336708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d40701f0 0x7f64d4160420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097d0 tx=0x7f64c402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59b6c28df46449ab server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.919+0000 7f64d8c4b640 1 -- 172.21.15.46:0/2592336708 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4035020 con 0x7f64d40701f0 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.919+0000 7f64d8c4b640 1 -- 172.21.15.46:0/2592336708 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c4031800 con 0x7f64d40701f0 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.919+0000 7f64d8c4b640 1 -- 172.21.15.46:0/2592336708 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4031bb0 con 0x7f64d40701f0 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 -- 172.21.15.46:0/2592336708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d40701f0 msgr2=0x7f64d4160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/2592336708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d40701f0 0x7f64d4160420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097d0 tx=0x7f64c402b8a0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 -- 172.21.15.46:0/2592336708 shutdown_connections 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/2592336708 >> v2:172.21.15.78:3300/0 conn(0x7f64d415fb30 0x7f64d40d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/2592336708 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d40701f0 0x7f64d4160420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 -- 172.21.15.46:0/2592336708 >> 172.21.15.46:0/2592336708 conn(0x7f64d40d2460 msgr2=0x7f64d40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 -- 172.21.15.46:0/2592336708 shutdown_connections 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.920+0000 7f64dbeb1640 1 -- 172.21.15.46:0/2592336708 wait complete. 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.921+0000 7f64dbeb1640 1 Processor -- start 2024-04-15T15:17:53.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.921+0000 7f64dbeb1640 1 -- start start 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64dbeb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 0x7f64d40ce940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64dbeb1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d409a9d0 0x7f64d40c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64dbeb1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d415fb30 0x7f64d406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64dbeb1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64d415ffc0 con 0x7f64d40701f0 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64dbeb1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4071050 con 0x7f64d409a9d0 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64dbeb1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f64d40b1c60 con 0x7f64d415fb30 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 0x7f64d40ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 0x7f64d40ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41902/0 (socket says 172.21.15.46:41902) 2024-04-15T15:17:53.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 -- 172.21.15.46:0/375150852 learned_addr learned my addr 172.21.15.46:0/375150852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64da44e640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d415fb30 0x7f64d406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d944c640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d409a9d0 0x7f64d40c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 -- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d415fb30 msgr2=0x7f64d406dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d415fb30 0x7f64d406dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 -- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d409a9d0 msgr2=0x7f64d40c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d409a9d0 0x7f64d40c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:53.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d9c4d640 1 -- 172.21.15.46:0/375150852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d40cd8a0 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.922+0000 7f64d944c640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d409a9d0 0x7f64d40c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.923+0000 7f64d9c4d640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 0x7f64d40ce940 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f64d000cb70 tx=0x7f64d000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.923+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0017070 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.923+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d000ac30 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.923+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d40cdb80 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.923+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0012400 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.923+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d40ce140 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.924+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d4071f70 con 0x7f64d40701f0 2024-04-15T15:17:53.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.924+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f64d001a070 con 0x7f64d40701f0 2024-04-15T15:17:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.925+0000 7f64ca7fc640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64b806c980 0x7f64b806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.925+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f64d008b930 con 0x7f64d40701f0 2024-04-15T15:17:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.925+0000 7f64d944c640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64b806c980 0x7f64b806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:53.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.925+0000 7f64d944c640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64b806c980 0x7f64b806ee40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f64c4036000 tx=0x7f64c403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:53.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:53.929+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64d4071f70 con 0x7f64d40701f0 2024-04-15T15:17:54.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.109+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f64d40689e0 con 0x7f64b806c980 2024-04-15T15:17:54.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.110+0000 7f64ca7fc640 1 -- 172.21.15.46:0/375150852 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13479 (secure 0 0 0) 0x7f64d40689e0 con 0x7f64b806c980 2024-04-15T15:17:54.111 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:54.112 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64b806c980 msgr2=0x7f64b806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64b806c980 0x7f64b806ee40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f64c4036000 tx=0x7f64c403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 msgr2=0x7f64d40ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 0x7f64d40ce940 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f64d000cb70 tx=0x7f64d000cf70 comp rx=0 tx=0).stop 2024-04-15T15:17:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 shutdown_connections 2024-04-15T15:17:54.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f64b806c980 0x7f64b806ee40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f64d415fb30 0x7f64d406dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d409a9d0 0x7f64d40c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 --2- 172.21.15.46:0/375150852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64d40701f0 0x7f64d40ce940 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 >> 172.21.15.46:0/375150852 conn(0x7f64d40d2460 msgr2=0x7f64d4161ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:54.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 shutdown_connections 2024-04-15T15:17:54.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.112+0000 7f64dbeb1640 1 -- 172.21.15.46:0/375150852 wait complete. 2024-04-15T15:17:54.125 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-15T15:17:52.386130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":238020,"kb_used_data":5196,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736700,"statfs":{"total":773094113280,"available":772850380800,"internally_reserved":0,"allocated":5320704,"data_stored":3002173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61070,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-128,"num_write":-92,"num_write_kb":-94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001392"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:49.420142+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:49.420142+0000","last_peered":"2024-04-15T15:17:49.420142+0000","last_clean":"2024-04-15T15:17:49.420142+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:49.420142+0000","last_undegraded":"2024-04-15T15:17:49.420142+0000","last_fullsized":"2024-04-15T15:17:49.420142+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542177,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542177,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27520,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1110016,"data_stored":816649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":816516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:54.126 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:17:54.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:54.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.229+0000 7fcda102c640 1 Processor -- start 2024-04-15T15:17:54.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.229+0000 7fcda102c640 1 -- start start 2024-04-15T15:17:54.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcda102c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c0701f0 0x7fcd9c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcda102c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd9c070720 con 0x7fcd9c09a9d0 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcda102c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd9c070a90 con 0x7fcd9c15fb30 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcda102c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcd9c09ba90 con 0x7fcd9c0701f0 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd9c09a9d0 0x7fcd9c09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59296/0 (socket says 172.21.15.46:59296) 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9ad9d640 1 -- 172.21.15.46:0/1906278300 learned_addr learned my addr 172.21.15.46:0/1906278300 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9259c640 1 --2- 172.21.15.46:0/1906278300 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c0701f0 0x7fcd9c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2285849457 0 0) 0x7fcd9c070720 con 0x7fcd9c09a9d0 2024-04-15T15:17:54.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd88003520 con 0x7fcd9c09a9d0 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206911344 0 0) 0x7fcd9c070a90 con 0x7fcd9c15fb30 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd9c070720 con 0x7fcd9c15fb30 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9259c640 1 -- 172.21.15.46:0/1906278300 >> v1:172.21.15.78:6789/0 conn(0x7fcd9c15fb30 legacy=0x7fcd9c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9259c640 1 -- 172.21.15.46:0/1906278300 >> v1:172.21.15.46:6789/0 conn(0x7fcd9c09a9d0 legacy=0x7fcd9c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.230+0000 7fcd9259c640 1 -- 172.21.15.46:0/1906278300 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9c070d70 con 0x7fcd9c0701f0 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.231+0000 7fcd9259c640 1 --2- 172.21.15.46:0/1906278300 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c0701f0 0x7fcd9c160420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009800 tx=0x7fcd8402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ca5c8423d6760c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:54.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.231+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84035020 con 0x7fcd9c0701f0 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.231+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd84031890 con 0x7fcd9c0701f0 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.231+0000 7fcd9a59c640 1 -- 172.21.15.46:0/1906278300 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd84031c60 con 0x7fcd9c0701f0 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.231+0000 7fcda102c640 1 -- 172.21.15.46:0/1906278300 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c0701f0 msgr2=0x7fcd9c160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.231+0000 7fcda102c640 1 --2- 172.21.15.46:0/1906278300 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c0701f0 0x7fcd9c160420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009800 tx=0x7fcd8402b930 comp rx=0 tx=0).stop 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.232+0000 7fcda102c640 1 -- 172.21.15.46:0/1906278300 shutdown_connections 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.232+0000 7fcda102c640 1 --2- 172.21.15.46:0/1906278300 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c0701f0 0x7fcd9c160420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.232+0000 7fcda102c640 1 -- 172.21.15.46:0/1906278300 >> 172.21.15.46:0/1906278300 conn(0x7fcd9c0d2460 msgr2=0x7fcd9c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:54.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.232+0000 7fcda102c640 1 -- 172.21.15.46:0/1906278300 shutdown_connections 2024-04-15T15:17:54.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.232+0000 7fcda102c640 1 -- 172.21.15.46:0/1906278300 wait complete. 2024-04-15T15:17:54.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.233+0000 7fcda102c640 1 Processor -- start 2024-04-15T15:17:54.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.233+0000 7fcda102c640 1 -- start start 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcda102c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 0x7fcd9c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcda102c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9c09a9d0 0x7fcd9c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcda102c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c15fb30 0x7fcd9c0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcda102c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c15ffc0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcda102c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fcd9c0d08c0 con 0x7fcd9c09a9d0 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcda102c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fcd9c0b1c60 con 0x7fcd9c15fb30 2024-04-15T15:17:54.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 0x7fcd9c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 0x7fcd9c157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41918/0 (socket says 172.21.15.46:41918) 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 -- 172.21.15.46:0/1092740596 learned_addr learned my addr 172.21.15.46:0/1092740596 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9b59e640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c15fb30 0x7fcd9c0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9259c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9c09a9d0 0x7fcd9c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 -- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c15fb30 msgr2=0x7fcd9c0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c15fb30 0x7fcd9c0b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 -- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9c09a9d0 msgr2=0x7fcd9c0b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9c09a9d0 0x7fcd9c0b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.234+0000 7fcd9ad9d640 1 -- 172.21.15.46:0/1092740596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd9c1577d0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.235+0000 7fcd9ad9d640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 0x7fcd9c157290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00c410 tx=0x7fcd8c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.235+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c017070 con 0x7fcd9c0701f0 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.235+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd8c005ce0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.235+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9c1e89f0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.235+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c0072d0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.235+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd9c0af910 con 0x7fcd9c0701f0 2024-04-15T15:17:54.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.236+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd9c0689e0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.236+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcd8c01a070 con 0x7fcd9c0701f0 2024-04-15T15:17:54.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.237+0000 7fcd93fff640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd7c06c980 0x7fcd7c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.237+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fcd8c08add0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.237+0000 7fcd9259c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd7c06c980 0x7fcd7c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.238+0000 7fcd9259c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd7c06c980 0x7fcd7c06ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcd84036000 tx=0x7fcd8403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:54.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.241+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd8c058ce0 con 0x7fcd9c0701f0 2024-04-15T15:17:54.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.422+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd9c0d3370 con 0x7fcd7c06c980 2024-04-15T15:17:54.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.423+0000 7fcd93fff640 1 -- 172.21.15.46:0/1092740596 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13479 (secure 0 0 0) 0x7fcd8404b4c0 con 0x7fcd7c06c980 2024-04-15T15:17:54.425 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:54.425 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:54.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd7c06c980 msgr2=0x7fcd7c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd7c06c980 0x7fcd7c06ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcd84036000 tx=0x7fcd8403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:54.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 msgr2=0x7fcd9c157290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 0x7fcd9c157290 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c00c410 tx=0x7fcd8c00c8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 shutdown_connections 2024-04-15T15:17:54.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fcd7c06c980 0x7fcd7c06ee40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fcd9c15fb30 0x7fcd9c0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd9c09a9d0 0x7fcd9c0b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 --2- 172.21.15.46:0/1092740596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd9c0701f0 0x7fcd9c157290 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 >> 172.21.15.46:0/1092740596 conn(0x7fcd9c0d2460 msgr2=0x7fcd9c1625b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:54.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 shutdown_connections 2024-04-15T15:17:54.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.425+0000 7fcda102c640 1 -- 172.21.15.46:0/1092740596 wait complete. 2024-04-15T15:17:54.438 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-04-15T15:17:54.386465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":238020,"kb_used_data":5196,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736700,"statfs":{"total":773094113280,"available":772850380800,"internally_reserved":0,"allocated":5320704,"data_stored":3002173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-61070,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-128,"num_write":-92,"num_write_kb":-94,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.972684"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:49.420142+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:49.420142+0000","last_peered":"2024-04-15T15:17:49.420142+0000","last_clean":"2024-04-15T15:17:49.420142+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:49.420142+0000","last_undegraded":"2024-04-15T15:17:49.420142+0000","last_fullsized":"2024-04-15T15:17:49.420142+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542177,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542177,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27520,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344320,"statfs":{"total":96636764160,"available":96608583680,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542176,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1110016,"data_stored":816649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31684,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":226148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1105920,"data_stored":816516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:54.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:54.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.542+0000 7f3c161a8640 1 Processor -- start 2024-04-15T15:17:54.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.542+0000 7f3c161a8640 1 -- start start 2024-04-15T15:17:54.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c161a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c161a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c100d1440 0x7f3c100b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c161a8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c100d2480 con 0x7f3c1015e790 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c161a8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3c100d2db0 con 0x7f3c100d1440 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c161a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3c100d2760 con 0x7f3c100d3240 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c0effd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c0ffff640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3c100d1440 0x7f3c100b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c0effd640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100d0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:57360/0 (socket says 172.21.15.46:57360) 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.543+0000 7f3c0effd640 1 -- 172.21.15.46:0/60471150 learned_addr learned my addr 172.21.15.46:0/60471150 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:54.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0ffff640 1 -- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 msgr2=0x7f3c100d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0ffff640 1 --2- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0ffff640 1 -- 172.21.15.46:0/60471150 >> v1:172.21.15.78:6789/0 conn(0x7f3c1015e790 legacy=0x7f3c100cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0ffff640 1 -- 172.21.15.46:0/60471150 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c100d2a40 con 0x7f3c100d1440 2024-04-15T15:17:54.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0effd640 1 --2- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0ffff640 1 --2- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c100b74f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009d90 tx=0x7f3c0802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f076b338400980a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.544+0000 7f3c0e7fc640 1 -- 172.21.15.46:0/60471150 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08039070 con 0x7f3c100d1440 2024-04-15T15:17:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c0e7fc640 1 -- 172.21.15.46:0/60471150 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c0802cc50 con 0x7f3c100d1440 2024-04-15T15:17:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c0e7fc640 1 -- 172.21.15.46:0/60471150 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08034440 con 0x7f3c100d1440 2024-04-15T15:17:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 -- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 msgr2=0x7f3c100b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 --2- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c100b74f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009d90 tx=0x7f3c0802bea0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 -- 172.21.15.46:0/60471150 shutdown_connections 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 --2- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c100b74f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 --2- 172.21.15.46:0/60471150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 -- 172.21.15.46:0/60471150 >> 172.21.15.46:0/60471150 conn(0x7f3c1008bf60 msgr2=0x7f3c100d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 -- 172.21.15.46:0/60471150 shutdown_connections 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.545+0000 7f3c161a8640 1 -- 172.21.15.46:0/60471150 wait complete. 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.546+0000 7f3c161a8640 1 Processor -- start 2024-04-15T15:17:54.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.546+0000 7f3c161a8640 1 -- start start 2024-04-15T15:17:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c161a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c101587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c161a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c161a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 0x7f3c100cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c161a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3c100d5180 con 0x7f3c1015e790 2024-04-15T15:17:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c161a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f3c100d0430 con 0x7f3c100d1440 2024-04-15T15:17:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c161a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3c100d5300 con 0x7f3c100d3240 2024-04-15T15:17:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c0ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 0x7f3c100cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c0ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 0x7f3c100cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41934/0 (socket says 172.21.15.46:41934) 2024-04-15T15:17:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c0ffff640 1 -- 172.21.15.46:0/527469771 learned_addr learned my addr 172.21.15.46:0/527469771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:54.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c0f7fe640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c101587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.547+0000 7f3c0effd640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c0ffff640 1 -- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 msgr2=0x7f3c100ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c0ffff640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c0ffff640 1 -- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 msgr2=0x7f3c101587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c0ffff640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c101587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c0ffff640 1 -- 172.21.15.46:0/527469771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c100cbc00 con 0x7f3c1015e790 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c0ffff640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 0x7f3c100cc100 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009eb0 tx=0x7f3c08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:54.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08039050 con 0x7f3c1015e790 2024-04-15T15:17:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c08045070 con 0x7f3c1015e790 2024-04-15T15:17:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c100c9f10 con 0x7f3c1015e790 2024-04-15T15:17:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08009450 con 0x7f3c1015e790 2024-04-15T15:17:54.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.548+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c100ce5f0 con 0x7f3c1015e790 2024-04-15T15:17:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.549+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c100689e0 con 0x7f3c1015e790 2024-04-15T15:17:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.550+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3c0803e610 con 0x7f3c1015e790 2024-04-15T15:17:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.550+0000 7f3c07fff640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3bf006c980 0x7f3bf006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:54.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.550+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f3c080ae360 con 0x7f3c1015e790 2024-04-15T15:17:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.550+0000 7f3c0f7fe640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3bf006c980 0x7f3bf006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.551+0000 7f3c0f7fe640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3bf006c980 0x7f3bf006ee40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c000016e0 tx=0x7f3c00009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:54.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.554+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c0807c300 con 0x7f3c1015e790 2024-04-15T15:17:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.742+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c100cbee0 con 0x7f3c1015e790 2024-04-15T15:17:54.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.743+0000 7f3c07fff640 1 -- 172.21.15.46:0/527469771 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+13371 (secure 0 0 0) 0x7f3c08080600 con 0x7f3c1015e790 2024-04-15T15:17:54.744 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:54.745 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":45,"fsid":"95f1e89e-6559-430a-8622-bd018dcc0afd","created":"2024-04-15T15:15:14.094940+0000","modified":"2024-04-15T15:17:50.415592+0000","last_up_change":"2024-04-15T15:15:25.499654+0000","last_in_change":"2024-04-15T15:15:16.367003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:15:26.367193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9d790af-808a-4ab7-8700-a7ccec98f3d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6818","nonce":2271822553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6822","nonce":2271822553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6827","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6829","nonce":2271822553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6823","nonce":2271822553},{"type":"v1","addr":"172.21.15.46:6825","nonce":2271822553}]},"public_addr":"172.21.15.46:6818/2271822553","cluster_addr":"172.21.15.46:6822/2271822553","heartbeat_back_addr":"172.21.15.46:6829/2271822553","heartbeat_front_addr":"172.21.15.46:6825/2271822553","state":["exists","up"]},{"osd":1,"uuid":"e6b51350-f913-4f16-ab9f-dbb94a6279ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6817","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6819","nonce":1258448605}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6821","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6824","nonce":1258448605}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6831","nonce":1258448605}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1258448605},{"type":"v1","addr":"172.21.15.46:6828","nonce":1258448605}]},"public_addr":"172.21.15.46:6819/1258448605","cluster_addr":"172.21.15.46:6824/1258448605","heartbeat_back_addr":"172.21.15.46:6831/1258448605","heartbeat_front_addr":"172.21.15.46:6828/1258448605","state":["exists","up"]},{"osd":2,"uuid":"04a46859-f5e6-441d-81ba-5245caffd22b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6801","nonce":1733872723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6803","nonce":1733872723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6807","nonce":1733872723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":1733872723},{"type":"v1","addr":"172.21.15.46:6805","nonce":1733872723}]},"public_addr":"172.21.15.46:6801/1733872723","cluster_addr":"172.21.15.46:6803/1733872723","heartbeat_back_addr":"172.21.15.46:6807/1733872723","heartbeat_front_addr":"172.21.15.46:6805/1733872723","state":["exists","up"]},{"osd":3,"uuid":"6a9a7eba-84d9-46fc-a7e1-8d933faaa2b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6809","nonce":3958614464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6811","nonce":3958614464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6815","nonce":3958614464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":3958614464},{"type":"v1","addr":"172.21.15.46:6813","nonce":3958614464}]},"public_addr":"172.21.15.46:6809/3958614464","cluster_addr":"172.21.15.46:6811/3958614464","heartbeat_back_addr":"172.21.15.46:6815/3958614464","heartbeat_front_addr":"172.21.15.46:6813/3958614464","state":["exists","up"]},{"osd":4,"uuid":"0192cc67-2b52-4c62-882f-5ccd13326bb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6801","nonce":3745610613}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6803","nonce":3745610613}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6807","nonce":3745610613}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":3745610613},{"type":"v1","addr":"172.21.15.78:6805","nonce":3745610613}]},"public_addr":"172.21.15.78:6801/3745610613","cluster_addr":"172.21.15.78:6803/3745610613","heartbeat_back_addr":"172.21.15.78:6807/3745610613","heartbeat_front_addr":"172.21.15.78:6805/3745610613","state":["exists","up"]},{"osd":5,"uuid":"821d1a3b-75b6-4850-9511-84b280133a6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6818","nonce":2916087727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6821","nonce":2916087727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6828","nonce":2916087727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6823","nonce":2916087727},{"type":"v1","addr":"172.21.15.78:6825","nonce":2916087727}]},"public_addr":"172.21.15.78:6818/2916087727","cluster_addr":"172.21.15.78:6821/2916087727","heartbeat_back_addr":"172.21.15.78:6828/2916087727","heartbeat_front_addr":"172.21.15.78:6825/2916087727","state":["exists","up"]},{"osd":6,"uuid":"38d4f697-6404-4d5a-96fe-d1e6dd1933be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6819","nonce":3189427077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6824","nonce":3189427077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6831","nonce":3189427077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6827","nonce":3189427077},{"type":"v1","addr":"172.21.15.78:6829","nonce":3189427077}]},"public_addr":"172.21.15.78:6819/3189427077","cluster_addr":"172.21.15.78:6824/3189427077","heartbeat_back_addr":"172.21.15.78:6831/3189427077","heartbeat_front_addr":"172.21.15.78:6829/3189427077","state":["exists","up"]},{"osd":7,"uuid":"9c0bac96-a0f3-4877-868b-dd55bad4a100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6809","nonce":563779653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6811","nonce":563779653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6815","nonce":563779653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":563779653},{"type":"v1","addr":"172.21.15.78:6813","nonce":563779653}]},"public_addr":"172.21.15.78:6809/563779653","cluster_addr":"172.21.15.78:6811/563779653","heartbeat_back_addr":"172.21.15.78:6815/563779653","heartbeat_front_addr":"172.21.15.78:6813/563779653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.550104+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.504082+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.430461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.522334+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.476752+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.527247+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.493853+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:15:21.491645+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/245964640":"2024-04-16T15:17:47.411051+0000","172.21.15.46:6835/964535021":"2024-04-16T15:17:30.625393+0000","172.21.15.46:6833/245964640":"2024-04-16T15:17:47.411051+0000","172.21.15.46:6833/964535021":"2024-04-16T15:17:30.625393+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3bf006c980 msgr2=0x7f3bf006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3bf006c980 0x7f3bf006ee40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c000016e0 tx=0x7f3c00009040 comp rx=0 tx=0).stop 2024-04-15T15:17:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 msgr2=0x7f3c100cc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 0x7f3c100cc100 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009eb0 tx=0x7f3c08004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 shutdown_connections 2024-04-15T15:17:54.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f3bf006c980 0x7f3bf006ee40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c1015e790 0x7f3c100cc100 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3c100d3240 0x7f3c100ce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 --2- 172.21.15.46:0/527469771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3c100d1440 0x7f3c101587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 >> 172.21.15.46:0/527469771 conn(0x7f3c1008bf60 msgr2=0x7f3c100b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 shutdown_connections 2024-04-15T15:17:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:54.745+0000 7f3c161a8640 1 -- 172.21.15.46:0/527469771 wait complete. 2024-04-15T15:17:55.758 INFO:tasks.ceph:Scrubbing osd.0 2024-04-15T15:17:55.759 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.879+0000 7f82a040c640 1 Processor -- start 2024-04-15T15:17:55.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.879+0000 7f82a040c640 1 -- start start 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.880+0000 7f82a040c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82980d09b0 0x7f82980d0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.880+0000 7f82a040c640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f82980ce060 0x7f829809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.880+0000 7f82a040c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.880+0000 7f82a040c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f82980cdbd0 con 0x7f82980d09b0 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.880+0000 7f82a040c640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f82980d12d0 con 0x7f82980ce060 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.880+0000 7f82a040c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f82980cd2a0 con 0x7f829809b920 2024-04-15T15:17:55.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82980d09b0 0x7f82980d0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f82980d09b0 0x7f82980d0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41938/0 (socket says 172.21.15.46:41938) 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 -- 172.21.15.46:0/3882861414 learned_addr learned my addr 172.21.15.46:0/3882861414 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e9a9640 1 --2- 172.21.15.46:0/3882861414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829d9a7640 1 --2- 172.21.15.46:0/3882861414 >> v2:172.21.15.78:3300/0 conn(0x7f82980ce060 0x7f829809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 -- 172.21.15.46:0/3882861414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 msgr2=0x7f82980b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 --2- 172.21.15.46:0/3882861414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 -- 172.21.15.46:0/3882861414 >> v2:172.21.15.78:3300/0 conn(0x7f82980ce060 msgr2=0x7f829809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:55.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 --2- 172.21.15.46:0/3882861414 >> v2:172.21.15.78:3300/0 conn(0x7f82980ce060 0x7f829809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.881+0000 7f829e1a8640 1 -- 172.21.15.46:0/3882861414 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82980cd580 con 0x7f82980d09b0 2024-04-15T15:17:55.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.882+0000 7f829e1a8640 1 --2- 172.21.15.46:0/3882861414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980d0d90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f82980b8480 tx=0x7f829402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93da03834d05719 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:55.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.882+0000 7f829d1a6640 1 -- 172.21.15.46:0/3882861414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294035020 con 0x7f82980d09b0 2024-04-15T15:17:55.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.882+0000 7f829d1a6640 1 -- 172.21.15.46:0/3882861414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8294031870 con 0x7f82980d09b0 2024-04-15T15:17:55.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.882+0000 7f829d1a6640 1 -- 172.21.15.46:0/3882861414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8294031c40 con 0x7f82980d09b0 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.882+0000 7f82a040c640 1 -- 172.21.15.46:0/3882861414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 msgr2=0x7f82980d0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 --2- 172.21.15.46:0/3882861414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980d0d90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f82980b8480 tx=0x7f829402b910 comp rx=0 tx=0).stop 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 -- 172.21.15.46:0/3882861414 shutdown_connections 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 --2- 172.21.15.46:0/3882861414 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 --2- 172.21.15.46:0/3882861414 >> v2:172.21.15.78:3300/0 conn(0x7f82980ce060 0x7f829809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 --2- 172.21.15.46:0/3882861414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980d0d90 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 -- 172.21.15.46:0/3882861414 >> 172.21.15.46:0/3882861414 conn(0x7f82980b3ec0 msgr2=0x7f82980b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:55.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 -- 172.21.15.46:0/3882861414 shutdown_connections 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.883+0000 7f82a040c640 1 -- 172.21.15.46:0/3882861414 wait complete. 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.884+0000 7f82a040c640 1 Processor -- start 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.884+0000 7f82a040c640 1 -- start start 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f82a040c640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f82a040c640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82980ce060 0x7f82980cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f82a040c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f82a040c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82980cd060 con 0x7f82980d09b0 2024-04-15T15:17:55.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f82a040c640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f82980ccee0 con 0x7f82980ce060 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f82a040c640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f82980d0110 con 0x7f829809b920 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41940/0 (socket says 172.21.15.46:41940) 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 -- 172.21.15.46:0/854445592 learned_addr learned my addr 172.21.15.46:0/854445592 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e1a8640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829d9a7640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82980ce060 0x7f82980cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 -- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 msgr2=0x7f82980c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f829809b920 0x7f82980c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 -- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82980ce060 msgr2=0x7f82980cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82980ce060 0x7f82980cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 -- 172.21.15.46:0/854445592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82980b7cd0 con 0x7f82980d09b0 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.885+0000 7f829e9a9640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980c7e00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8290007d20 tx=0x7f829000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.886+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290014420 con 0x7f82980d09b0 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.886+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8290019070 con 0x7f82980d09b0 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.886+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290014820 con 0x7f82980d09b0 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.886+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82980c7640 con 0x7f82980d09b0 2024-04-15T15:17:55.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.886+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82980c78e0 con 0x7f82980d09b0 2024-04-15T15:17:55.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.887+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8298068620 con 0x7f82980d09b0 2024-04-15T15:17:55.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.887+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f82900149c0 con 0x7f82980d09b0 2024-04-15T15:17:55.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f828affd640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f827c06c980 0x7f827c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f829008cfd0 con 0x7f82980d09b0 2024-04-15T15:17:55.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f828affd640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7f827c072ec0 0x7f827c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:55.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 --> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8290014e50 con 0x7f827c072ec0 2024-04-15T15:17:55.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f829008d4b0 con 0x7f82980d09b0 2024-04-15T15:17:55.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f829e1a8640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f827c06c980 0x7f827c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f829d9a7640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7f827c072ec0 0x7f827c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:55.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.888+0000 7f829d9a7640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7f827c072ec0 0x7f827c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:55.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.889+0000 7f829e1a8640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f827c06c980 0x7f827c06ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8294036000 tx=0x7f829403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:55.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.889+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== osd.0 v2:172.21.15.46:6816/2271822553 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8290014e50 con 0x7f827c072ec0 2024-04-15T15:17:55.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.902+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 --> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f82980690f0 con 0x7f827c072ec0 2024-04-15T15:17:55.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f828affd640 1 -- 172.21.15.46:0/854445592 <== osd.0 v2:172.21.15.46:6816/2271822553 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f82980690f0 con 0x7f827c072ec0 2024-04-15T15:17:55.905 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:55.905 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:55.905 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7f827c072ec0 msgr2=0x7f827c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:55.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:6816/2271822553,v1:172.21.15.46:6818/2271822553] conn(0x7f827c072ec0 0x7f827c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f827c06c980 msgr2=0x7f827c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f827c06c980 0x7f827c06ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8294036000 tx=0x7f829403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 msgr2=0x7f82980c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 --2- 172.21.15.46:0/854445592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82980d09b0 0x7f82980c7e00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8290007d20 tx=0x7f829000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f829e9a9640 1 -- 172.21.15.46:0/854445592 reap_dead start 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 shutdown_connections 2024-04-15T15:17:55.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 >> 172.21.15.46:0/854445592 conn(0x7f82980b3ec0 msgr2=0x7f82980b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 shutdown_connections 2024-04-15T15:17:55.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:55.903+0000 7f82a040c640 1 -- 172.21.15.46:0/854445592 wait complete. 2024-04-15T15:17:55.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-15T15:17:56.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.023+0000 7f7f3fb29640 1 Processor -- start 2024-04-15T15:17:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.023+0000 7f7f3fb29640 1 -- start start 2024-04-15T15:17:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3fb29640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f7f3815e700 0x7f7f380b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3fb29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f380d2360 0x7f7f380d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3fb29640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f380b47a0 con 0x7f7f3815eae0 2024-04-15T15:17:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3fb29640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f7f380b53b0 con 0x7f7f380d2360 2024-04-15T15:17:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3fb29640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f7f380b4a80 con 0x7f7f3815e700 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d0c4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f380d2360 0x7f7f380d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3e0c6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7f3815eae0 0x7f7f380cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59310/0 (socket says 172.21.15.46:59310) 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3e0c6640 1 -- 172.21.15.46:0/2174358655 learned_addr learned my addr 172.21.15.46:0/2174358655 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d8c5640 1 --2- 172.21.15.46:0/2174358655 >> v2:172.21.15.78:3300/0 conn(0x7f7f3815e700 0x7f7f380b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/2174358655 >> v2:172.21.15.78:3300/0 conn(0x7f7f3815e700 msgr2=0x7f7f380b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d0c4640 1 --2- 172.21.15.46:0/2174358655 >> v2:172.21.15.78:3300/0 conn(0x7f7f3815e700 0x7f7f380b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/2174358655 >> v1:172.21.15.46:6789/0 conn(0x7f7f3815eae0 legacy=0x7f7f380cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/2174358655 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f380b4d60 con 0x7f7f380d2360 2024-04-15T15:17:56.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.024+0000 7f7f3d8c5640 1 --2- 172.21.15.46:0/2174358655 >> v2:172.21.15.78:3300/0 conn(0x7f7f3815e700 0x7f7f380b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.025+0000 7f7f3d0c4640 1 --2- 172.21.15.46:0/2174358655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f380d2360 0x7f7f380d2b60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7f3806c6c0 tx=0x7f7f2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7933e130153c6c71 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.025+0000 7f7f3c8c3640 1 -- 172.21.15.46:0/2174358655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28035020 con 0x7f7f380d2360 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.025+0000 7f7f3c8c3640 1 -- 172.21.15.46:0/2174358655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f28031830 con 0x7f7f380d2360 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.025+0000 7f7f3c8c3640 1 -- 172.21.15.46:0/2174358655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28031c00 con 0x7f7f380d2360 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 -- 172.21.15.46:0/2174358655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f380d2360 msgr2=0x7f7f380d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/2174358655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f380d2360 0x7f7f380d2b60 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f7f3806c6c0 tx=0x7f7f2802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 -- 172.21.15.46:0/2174358655 shutdown_connections 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/2174358655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f380d2360 0x7f7f380d2b60 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/2174358655 >> v2:172.21.15.78:3300/0 conn(0x7f7f3815e700 0x7f7f380b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 -- 172.21.15.46:0/2174358655 >> 172.21.15.46:0/2174358655 conn(0x7f7f3808bf60 msgr2=0x7f7f380d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 -- 172.21.15.46:0/2174358655 shutdown_connections 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.026+0000 7f7f3fb29640 1 -- 172.21.15.46:0/2174358655 wait complete. 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.027+0000 7f7f3fb29640 1 Processor -- start 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.027+0000 7f7f3fb29640 1 -- start start 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3fb29640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f380d2360 0x7f7f3815ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3fb29640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 0x7f7f38151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3fb29640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7f3815eae0 0x7f7f380b91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3fb29640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f380b5d80 con 0x7f7f3815e700 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3fb29640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f7f380b4450 con 0x7f7f380d2360 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3fb29640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f7f380d1d40 con 0x7f7f3815eae0 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 0x7f7f38151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 0x7f7f38151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41958/0 (socket says 172.21.15.46:41958) 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/1647496962 learned_addr learned my addr 172.21.15.46:0/1647496962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d8c5640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f380d2360 0x7f7f3815ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3e0c6640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7f3815eae0 0x7f7f380b91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7f3815eae0 msgr2=0x7f7f380b91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7f3815eae0 0x7f7f380b91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f380d2360 msgr2=0x7f7f3815ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f380d2360 0x7f7f3815ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 -- 172.21.15.46:0/1647496962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f380ce740 con 0x7f7f3815e700 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d8c5640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f380d2360 0x7f7f3815ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.028+0000 7f7f3d0c4640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 0x7f7f38151af0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000c00 tx=0x7f7f280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.029+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28041070 con 0x7f7f3815e700 2024-04-15T15:17:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.029+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f2803a950 con 0x7f7f3815e700 2024-04-15T15:17:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.029+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f380b87e0 con 0x7f7f3815e700 2024-04-15T15:17:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.029+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28035040 con 0x7f7f3815e700 2024-04-15T15:17:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.029+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f380cc4e0 con 0x7f7f3815e700 2024-04-15T15:17:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.030+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f380689e0 con 0x7f7f3815e700 2024-04-15T15:17:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.030+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7f28038020 con 0x7f7f3815e700 2024-04-15T15:17:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.030+0000 7f7f2e7fc640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7f1c06c980 0x7f7f1c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.031+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f7f280ad4d0 con 0x7f7f3815e700 2024-04-15T15:17:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.031+0000 7f7f3d8c5640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7f1c06c980 0x7f7f1c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.031+0000 7f7f3d8c5640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7f1c06c980 0x7f7f1c06ee40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7f380d11b0 tx=0x7f7f34008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.039+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f2807ba30 con 0x7f7f3815e700 2024-04-15T15:17:56.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.225+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7f380d11e0 con 0x7f7f1c06c980 2024-04-15T15:17:56.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.226+0000 7f7f2e7fc640 1 -- 172.21.15.46:0/1647496962 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7f380d11e0 con 0x7f7f1c06c980 2024-04-15T15:17:56.228 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2024-04-15T15:17:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7f1c06c980 msgr2=0x7f7f1c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7f1c06c980 0x7f7f1c06ee40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7f380d11b0 tx=0x7f7f34008040 comp rx=0 tx=0).stop 2024-04-15T15:17:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 msgr2=0x7f7f38151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 0x7f7f38151af0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000c00 tx=0x7f7f280046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 shutdown_connections 2024-04-15T15:17:56.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f7f1c06c980 0x7f7f1c06ee40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f7f3815eae0 0x7f7f380b91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f3815e700 0x7f7f38151af0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 --2- 172.21.15.46:0/1647496962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f380d2360 0x7f7f3815ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 >> 172.21.15.46:0/1647496962 conn(0x7f7f3808bf60 msgr2=0x7f7f380d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 shutdown_connections 2024-04-15T15:17:56.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.228+0000 7f7f3fb29640 1 -- 172.21.15.46:0/1647496962 wait complete. 2024-04-15T15:17:56.241 INFO:tasks.ceph:Scrubbing osd.1 2024-04-15T15:17:56.241 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:56.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.349+0000 7fd2a5d9d640 1 Processor -- start 2024-04-15T15:17:56.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.349+0000 7fd2a5d9d640 1 -- start start 2024-04-15T15:17:56.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.350+0000 7fd2a5d9d640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd2a00ce770 0x7fd2a00cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.350+0000 7fd2a5d9d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2a00b3610 con 0x7fd2a00b3160 2024-04-15T15:17:56.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.350+0000 7fd2a5d9d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2a00b38d0 con 0x7fd2a00d0470 2024-04-15T15:17:56.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.350+0000 7fd2a5d9d640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd2a00b3bb0 con 0x7fd2a00ce770 2024-04-15T15:17:56.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.350+0000 7fd29effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd2a00b3160 0x7fd2a015c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59326/0 (socket says 172.21.15.46:59326) 2024-04-15T15:17:56.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.350+0000 7fd29effd640 1 -- 172.21.15.46:0/2002385794 learned_addr learned my addr 172.21.15.46:0/2002385794 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1134929758 0 0) 0x7fd2a00b3610 con 0x7fd2a00b3160 2024-04-15T15:17:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29f7fe640 1 --2- 172.21.15.46:0/2002385794 >> v2:172.21.15.78:3300/0 conn(0x7fd2a00ce770 0x7fd2a00cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd288003520 con 0x7fd2a00b3160 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4043062901 0 0) 0x7fd288003520 con 0x7fd2a00b3160 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 >> v2:172.21.15.78:3300/0 conn(0x7fd2a00ce770 msgr2=0x7fd2a00cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 --2- 172.21.15.46:0/2002385794 >> v2:172.21.15.78:3300/0 conn(0x7fd2a00ce770 0x7fd2a00cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 >> v1:172.21.15.78:6789/0 conn(0x7fd2a00d0470 legacy=0x7fd2a00ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a00b3e90 con 0x7fd2a00b3160 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fd294001670 con 0x7fd2a00b3160 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 >> v1:172.21.15.46:6789/0 conn(0x7fd2a00b3160 legacy=0x7fd2a015c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2880039b0 0x7fd288003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd288004800 0x7fd288004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2880051c0 0x7fd28800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a00b3bb0 con 0x7fd288004800 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd28800c160 con 0x7fd2880051c0 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd28800c360 con 0x7fd2880039b0 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29effd640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd288004800 0x7fd288004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29f7fe640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2880039b0 0x7fd288003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29ffff640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2880051c0 0x7fd28800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29effd640 1 -- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2880039b0 msgr2=0x7fd288003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29effd640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2880039b0 0x7fd288003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29effd640 1 -- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2880051c0 msgr2=0x7fd28800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29effd640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2880051c0 0x7fd28800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29effd640 1 -- 172.21.15.46:0/2002385794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd288003520 con 0x7fd288004800 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29ffff640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2880051c0 0x7fd28800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.351+0000 7fd29ffff640 1 -- 172.21.15.46:0/2002385794 reap_dead start 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.352+0000 7fd29effd640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd288004800 0x7fd288004c80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c240 tx=0x7fd288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22a34f2b94ebd40 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.352+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294007cf0 con 0x7fd288004800 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.352+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd294030070 con 0x7fd288004800 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.352+0000 7fd29e7fc640 1 -- 172.21.15.46:0/2002385794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294005d70 con 0x7fd288004800 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/2002385794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd288004800 msgr2=0x7fd288004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd288004800 0x7fd288004c80 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c240 tx=0x7fd288005830 comp rx=0 tx=0).stop 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/2002385794 shutdown_connections 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 --2- 172.21.15.46:0/2002385794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd288004800 0x7fd288004c80 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/2002385794 >> 172.21.15.46:0/2002385794 conn(0x7fd2a006bbc0 msgr2=0x7fd2a006bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/2002385794 shutdown_connections 2024-04-15T15:17:56.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.353+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/2002385794 wait complete. 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.354+0000 7fd2a5d9d640 1 Processor -- start 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.354+0000 7fd2a5d9d640 1 -- start start 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd2a5d9d640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288004800 0x7fd2a015c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd2a5d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a0148790 0x7fd2a01f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd2a5d9d640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2a01f5390 0x7fd2a015a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd2a5d9d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a00d0900 con 0x7fd2a0148790 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd2a5d9d640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a00ceb50 con 0x7fd288004800 2024-04-15T15:17:56.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd2a5d9d640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd2a00d0a80 con 0x7fd2a01f5390 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd29effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a0148790 0x7fd2a01f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd29effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a0148790 0x7fd2a01f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41974/0 (socket says 172.21.15.46:41974) 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd29effd640 1 -- 172.21.15.46:0/1551490127 learned_addr learned my addr 172.21.15.46:0/1551490127 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.355+0000 7fd29f7fe640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288004800 0x7fd2a015c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29ffff640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2a01f5390 0x7fd2a015a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29effd640 1 -- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2a01f5390 msgr2=0x7fd2a015a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29effd640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2a01f5390 0x7fd2a015a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29effd640 1 -- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288004800 msgr2=0x7fd2a015c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29effd640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288004800 0x7fd2a015c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29effd640 1 -- 172.21.15.46:0/1551490127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a015ced0 con 0x7fd2a0148790 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29f7fe640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd288004800 0x7fd2a015c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29ffff640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd2a01f5390 0x7fd2a015a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd29effd640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a0148790 0x7fd2a01f4e50 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c240 tx=0x7fd2a00b3aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294046070 con 0x7fd2a0148790 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd294000ed0 con 0x7fd2a0148790 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a01f5980 con 0x7fd2a0148790 2024-04-15T15:17:56.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294004e10 con 0x7fd2a0148790 2024-04-15T15:17:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.356+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a015c190 con 0x7fd2a0148790 2024-04-15T15:17:56.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.357+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2a00689e0 con 0x7fd2a0148790 2024-04-15T15:17:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.357+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd294007070 con 0x7fd2a0148790 2024-04-15T15:17:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd287fff640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd28c06c980 0x7fd28c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fd2940ad160 con 0x7fd2a0148790 2024-04-15T15:17:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd287fff640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fd28c072ec0 0x7fd28c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 --> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd29400bf30 con 0x7fd28c072ec0 2024-04-15T15:17:56.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fd2940ad640 con 0x7fd2a0148790 2024-04-15T15:17:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd29ffff640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fd28c072ec0 0x7fd28c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.358+0000 7fd29f7fe640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd28c06c980 0x7fd28c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.359+0000 7fd29ffff640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fd28c072ec0 0x7fd28c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.359+0000 7fd29f7fe640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd28c06c980 0x7fd28c06ee40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd290009b30 tx=0x7fd290008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.360+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== osd.1 v2:172.21.15.46:6817/1258448605 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd29400bf30 con 0x7fd28c072ec0 2024-04-15T15:17:56.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.373+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 --> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd2a00b3610 con 0x7fd28c072ec0 2024-04-15T15:17:56.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.373+0000 7fd287fff640 1 -- 172.21.15.46:0/1551490127 <== osd.1 v2:172.21.15.46:6817/1258448605 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd2a00b3610 con 0x7fd28c072ec0 2024-04-15T15:17:56.375 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:56.376 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:56.376 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:56.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.373+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fd28c072ec0 msgr2=0x7fd28c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.373+0000 7fd2a5d9d640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:6817/1258448605,v1:172.21.15.46:6819/1258448605] conn(0x7fd28c072ec0 0x7fd28c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd28c06c980 msgr2=0x7fd28c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd28c06c980 0x7fd28c06ee40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd290009b30 tx=0x7fd290008040 comp rx=0 tx=0).stop 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a0148790 msgr2=0x7fd2a01f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 --2- 172.21.15.46:0/1551490127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2a0148790 0x7fd2a01f4e50 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c240 tx=0x7fd2a00b3aa0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd29ffff640 1 -- 172.21.15.46:0/1551490127 reap_dead start 2024-04-15T15:17:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 shutdown_connections 2024-04-15T15:17:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 >> 172.21.15.46:0/1551490127 conn(0x7fd2a006bbc0 msgr2=0x7fd2a00c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 shutdown_connections 2024-04-15T15:17:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.374+0000 7fd2a5d9d640 1 -- 172.21.15.46:0/1551490127 wait complete. 2024-04-15T15:17:56.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-15T15:17:56.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.492+0000 7f0b39ad9640 1 Processor -- start 2024-04-15T15:17:56.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.492+0000 7f0b39ad9640 1 -- start start 2024-04-15T15:17:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b39ad9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409a9d0 0x7f0b3409adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b39ad9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b3415ebb0 0x7f0b3406cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b39ad9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b3409b2f0 con 0x7f0b3409b780 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b39ad9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0b340cd8c0 con 0x7f0b3415ebb0 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b39ad9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b340ced60 con 0x7f0b3409a9d0 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b32ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b3415ebb0 0x7f0b3406cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b33fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0b3409b780 0x7f0b340b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59342/0 (socket says 172.21.15.46:59342) 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b33fff640 1 -- 172.21.15.46:0/2507914837 learned_addr learned my addr 172.21.15.46:0/2507914837 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b337fe640 1 --2- 172.21.15.46:0/2507914837 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409a9d0 0x7f0b3409adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b32ffd640 1 -- 172.21.15.46:0/2507914837 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409a9d0 msgr2=0x7f0b3409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b32ffd640 1 --2- 172.21.15.46:0/2507914837 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409a9d0 0x7f0b3409adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b32ffd640 1 -- 172.21.15.46:0/2507914837 >> v1:172.21.15.46:6789/0 conn(0x7f0b3409b780 legacy=0x7f0b340b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b327fc640 1 -- 172.21.15.46:0/2507914837 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724733775 0 0) 0x7f0b3409b2f0 con 0x7f0b3409b780 2024-04-15T15:17:56.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.493+0000 7f0b32ffd640 1 -- 172.21.15.46:0/2507914837 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b340cccb0 con 0x7f0b3415ebb0 2024-04-15T15:17:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.494+0000 7f0b32ffd640 1 --2- 172.21.15.46:0/2507914837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3415ebb0 0x7f0b3406cd00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0b240097a0 tx=0x7f0b2402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287e870873ad0ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b327fc640 1 -- 172.21.15.46:0/2507914837 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24035020 con 0x7f0b3415ebb0 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b327fc640 1 -- 172.21.15.46:0/2507914837 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b24031830 con 0x7f0b3415ebb0 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b327fc640 1 -- 172.21.15.46:0/2507914837 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24031c00 con 0x7f0b3415ebb0 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 -- 172.21.15.46:0/2507914837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3415ebb0 msgr2=0x7f0b3406cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/2507914837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3415ebb0 0x7f0b3406cd00 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0b240097a0 tx=0x7f0b2402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 -- 172.21.15.46:0/2507914837 shutdown_connections 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/2507914837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3415ebb0 0x7f0b3406cd00 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/2507914837 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409a9d0 0x7f0b3409adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 -- 172.21.15.46:0/2507914837 >> 172.21.15.46:0/2507914837 conn(0x7f0b340b3fb0 msgr2=0x7f0b340b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 -- 172.21.15.46:0/2507914837 shutdown_connections 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.495+0000 7f0b39ad9640 1 -- 172.21.15.46:0/2507914837 wait complete. 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.496+0000 7f0b39ad9640 1 Processor -- start 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 -- start start 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 0x7f0b341518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409b780 0x7f0b340c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b340c89f0 0x7f0b340b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b340cd550 con 0x7f0b3409a9d0 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0b340cd270 con 0x7f0b340c89f0 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.497+0000 7f0b39ad9640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0b3406d240 con 0x7f0b3409b780 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 0x7f0b341518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 0x7f0b341518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41996/0 (socket says 172.21.15.46:41996) 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 -- 172.21.15.46:0/295130169 learned_addr learned my addr 172.21.15.46:0/295130169 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b32ffd640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409b780 0x7f0b340c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b33fff640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b340c89f0 0x7f0b340b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 -- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409b780 msgr2=0x7f0b340c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409b780 0x7f0b340c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 -- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b340c89f0 msgr2=0x7f0b340b6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b340c89f0 0x7f0b340b6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 -- 172.21.15.46:0/295130169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b340c6aa0 con 0x7f0b3409a9d0 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b33fff640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b340c89f0 0x7f0b340b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b337fe640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 0x7f0b341518e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00d800 tx=0x7f0b1c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00b6c0 con 0x7f0b3409a9d0 2024-04-15T15:17:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b1c019070 con 0x7f0b3409a9d0 2024-04-15T15:17:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c00ba80 con 0x7f0b3409a9d0 2024-04-15T15:17:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b340c6d80 con 0x7f0b3409a9d0 2024-04-15T15:17:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.498+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b340c7340 con 0x7f0b3409a9d0 2024-04-15T15:17:56.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.499+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b340690f0 con 0x7f0b3409a9d0 2024-04-15T15:17:56.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.500+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0b1c00bc20 con 0x7f0b3409a9d0 2024-04-15T15:17:56.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.500+0000 7f0b2bfff640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b1406c9b0 0x7f0b1406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.500+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f0b1c08bf10 con 0x7f0b3409a9d0 2024-04-15T15:17:56.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.501+0000 7f0b32ffd640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b1406c9b0 0x7f0b1406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.501+0000 7f0b32ffd640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b1406c9b0 0x7f0b1406ee70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0b24036000 tx=0x7f0b2403c040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.509+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b1c05a3e0 con 0x7f0b3409a9d0 2024-04-15T15:17:56.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.693+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f0b34075330 con 0x7f0b1406c9b0 2024-04-15T15:17:56.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.693+0000 7f0b2bfff640 1 -- 172.21.15.46:0/295130169 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0b34075330 con 0x7f0b1406c9b0 2024-04-15T15:17:56.695 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2024-04-15T15:17:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b1406c9b0 msgr2=0x7f0b1406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b1406c9b0 0x7f0b1406ee70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0b24036000 tx=0x7f0b2403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:56.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 msgr2=0x7f0b341518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 0x7f0b341518e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00d800 tx=0x7f0b1c00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 shutdown_connections 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0b1406c9b0 0x7f0b1406ee70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0b340c89f0 0x7f0b340b6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0b3409b780 0x7f0b340c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 --2- 172.21.15.46:0/295130169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b3409a9d0 0x7f0b341518e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 >> 172.21.15.46:0/295130169 conn(0x7f0b340b3fb0 msgr2=0x7f0b34070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 shutdown_connections 2024-04-15T15:17:56.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.695+0000 7f0b39ad9640 1 -- 172.21.15.46:0/295130169 wait complete. 2024-04-15T15:17:56.708 INFO:tasks.ceph:Scrubbing osd.2 2024-04-15T15:17:56.708 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:56.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.815+0000 7fd712c77640 1 Processor -- start 2024-04-15T15:17:56.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.815+0000 7fd712c77640 1 -- start start 2024-04-15T15:17:56.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd712c77640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fd70c0d0c30 0x7fd70c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd712c77640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd70c0b21b0 0x7fd70c0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd712c77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd70c0d1010 con 0x7fd70c06b9e0 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd712c77640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fd70c0b2b60 con 0x7fd70c0b21b0 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd712c77640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fd70c0ce0b0 con 0x7fd70c0d0c30 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd70c0b21b0 0x7fd70c0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd70c0b21b0 0x7fd70c0b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42000/0 (socket says 172.21.15.46:42000) 2024-04-15T15:17:56.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 -- 172.21.15.46:0/1820099610 learned_addr learned my addr 172.21.15.46:0/1820099610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:56.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd710a13640 1 --2- 172.21.15.46:0/1820099610 >> v2:172.21.15.78:3300/0 conn(0x7fd70c0d0c30 0x7fd70c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 -- 172.21.15.46:0/1820099610 >> v2:172.21.15.78:3300/0 conn(0x7fd70c0d0c30 msgr2=0x7fd70c0cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 --2- 172.21.15.46:0/1820099610 >> v2:172.21.15.78:3300/0 conn(0x7fd70c0d0c30 0x7fd70c0cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 -- 172.21.15.46:0/1820099610 >> v1:172.21.15.78:6789/0 conn(0x7fd70c06b9e0 legacy=0x7fd70c0ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd711214640 1 -- 172.21.15.46:0/1820099610 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c06c1a0 con 0x7fd70c0b21b0 2024-04-15T15:17:56.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.816+0000 7fd710a13640 1 --2- 172.21.15.46:0/1820099610 >> v2:172.21.15.78:3300/0 conn(0x7fd70c0d0c30 0x7fd70c0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:56.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.817+0000 7fd711214640 1 --2- 172.21.15.46:0/1820099610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0b21b0 0x7fd70c0b2620 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0d1220 tx=0x7fd70402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efe3c7f8e735e72 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd7037fe640 1 -- 172.21.15.46:0/1820099610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70402ca80 con 0x7fd70c0b21b0 2024-04-15T15:17:56.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd7037fe640 1 -- 172.21.15.46:0/1820099610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70402cc20 con 0x7fd70c0b21b0 2024-04-15T15:17:56.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd7037fe640 1 -- 172.21.15.46:0/1820099610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7040343f0 con 0x7fd70c0b21b0 2024-04-15T15:17:56.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 -- 172.21.15.46:0/1820099610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0b21b0 msgr2=0x7fd70c0b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 --2- 172.21.15.46:0/1820099610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0b21b0 0x7fd70c0b2620 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0d1220 tx=0x7fd70402bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 -- 172.21.15.46:0/1820099610 shutdown_connections 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 --2- 172.21.15.46:0/1820099610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0b21b0 0x7fd70c0b2620 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 --2- 172.21.15.46:0/1820099610 >> v2:172.21.15.78:3300/0 conn(0x7fd70c0d0c30 0x7fd70c0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 -- 172.21.15.46:0/1820099610 >> 172.21.15.46:0/1820099610 conn(0x7fd70c0b2e70 msgr2=0x7fd70c0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 -- 172.21.15.46:0/1820099610 shutdown_connections 2024-04-15T15:17:56.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.818+0000 7fd712c77640 1 -- 172.21.15.46:0/1820099610 wait complete. 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.820+0000 7fd712c77640 1 Processor -- start 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.820+0000 7fd712c77640 1 -- start start 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd712c77640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd70c06b9e0 0x7fd70c153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd712c77640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd70c0b21b0 0x7fd70c09bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd712c77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0d0c30 0x7fd70c0af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd712c77640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c06be20 con 0x7fd70c0d0c30 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd712c77640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c0cdd40 con 0x7fd70c0b21b0 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd712c77640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fd70c0b0b10 con 0x7fd70c06b9e0 2024-04-15T15:17:56.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0d0c30 0x7fd70c0af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0d0c30 0x7fd70c0af7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42008/0 (socket says 172.21.15.46:42008) 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 -- 172.21.15.46:0/4239751967 learned_addr learned my addr 172.21.15.46:0/4239751967 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd710a13640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd70c06b9e0 0x7fd70c153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd703fff640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd70c0b21b0 0x7fd70c09bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 -- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd70c06b9e0 msgr2=0x7fd70c153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fd70c06b9e0 0x7fd70c153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 -- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd70c0b21b0 msgr2=0x7fd70c09bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd70c0b21b0 0x7fd70c09bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 -- 172.21.15.46:0/4239751967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c09c920 con 0x7fd70c0d0c30 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.822+0000 7fd711214640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0d0c30 0x7fd70c0af7a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd70402c4f0 tx=0x7fd704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.823+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704043020 con 0x7fd70c0d0c30 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.823+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70403e450 con 0x7fd70c0d0c30 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.823+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c09cc00 con 0x7fd70c0d0c30 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.823+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70403e820 con 0x7fd70c0d0c30 2024-04-15T15:17:56.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.823+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c09d1c0 con 0x7fd70c0d0c30 2024-04-15T15:17:56.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.823+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd70c0690f0 con 0x7fd70c0d0c30 2024-04-15T15:17:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.824+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd704046020 con 0x7fd70c0d0c30 2024-04-15T15:17:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd7017fa640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd6f806c980 0x7fd6f806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fd7040ae990 con 0x7fd70c0d0c30 2024-04-15T15:17:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd7017fa640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fd6f8072ec0 0x7fd6f8075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 --> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd70403ae50 con 0x7fd6f8072ec0 2024-04-15T15:17:56.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fd7040aed40 con 0x7fd70c0d0c30 2024-04-15T15:17:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd710a13640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd6f806c980 0x7fd6f806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd703fff640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fd6f8072ec0 0x7fd6f8075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd703fff640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fd6f8072ec0 0x7fd6f8075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.825+0000 7fd710a13640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd6f806c980 0x7fd6f806ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd70800aac0 tx=0x7fd708008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.826+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== osd.2 v2:172.21.15.46:6800/1733872723 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd70403ae50 con 0x7fd6f8072ec0 2024-04-15T15:17:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.839+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 --> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd70c06f8c0 con 0x7fd6f8072ec0 2024-04-15T15:17:56.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.839+0000 7fd7017fa640 1 -- 172.21.15.46:0/4239751967 <== osd.2 v2:172.21.15.46:6800/1733872723 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd70c06f8c0 con 0x7fd6f8072ec0 2024-04-15T15:17:56.841 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:56.841 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:56.842 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fd6f8072ec0 msgr2=0x7fd6f8075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:6800/1733872723,v1:172.21.15.46:6801/1733872723] conn(0x7fd6f8072ec0 0x7fd6f8075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd6f806c980 msgr2=0x7fd6f806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fd6f806c980 0x7fd6f806ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd70800aac0 tx=0x7fd708008040 comp rx=0 tx=0).stop 2024-04-15T15:17:56.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0d0c30 msgr2=0x7fd70c0af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 --2- 172.21.15.46:0/4239751967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c0d0c30 0x7fd70c0af7a0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd70402c4f0 tx=0x7fd704004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd711214640 1 -- 172.21.15.46:0/4239751967 reap_dead start 2024-04-15T15:17:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 shutdown_connections 2024-04-15T15:17:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 >> 172.21.15.46:0/4239751967 conn(0x7fd70c0b2e70 msgr2=0x7fd70c0700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 shutdown_connections 2024-04-15T15:17:56.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.840+0000 7fd712c77640 1 -- 172.21.15.46:0/4239751967 wait complete. 2024-04-15T15:17:56.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-15T15:17:56.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.956+0000 7f66b8488640 1 Processor -- start 2024-04-15T15:17:56.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.956+0000 7f66b8488640 1 -- start start 2024-04-15T15:17:56.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b8488640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f66b00d2360 0x7f66b00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b8488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f66b00d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b8488640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b00b47a0 con 0x7f66b015e700 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b8488640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f66b00b53b0 con 0x7f66b00d2360 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b8488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f66b00b4a80 con 0x7f66b015eae0 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b6224640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66b015e700 0x7f66b00b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59344/0 (socket says 172.21.15.46:59344) 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b6224640 1 -- 172.21.15.46:0/915609339 learned_addr learned my addr 172.21.15.46:0/915609339 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b6a25640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f66b00d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b5a23640 1 --2- 172.21.15.46:0/915609339 >> v2:172.21.15.78:3300/0 conn(0x7f66b00d2360 0x7f66b00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.957+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280059049 0 0) 0x7f66b00b47a0 con 0x7f66b015e700 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f669c003520 con 0x7f66b015e700 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3005602367 0 0) 0x7f669c003520 con 0x7f66b015e700 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 msgr2=0x7f66b00d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f66b00d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 >> v2:172.21.15.78:3300/0 conn(0x7f66b00d2360 msgr2=0x7f66b00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 --2- 172.21.15.46:0/915609339 >> v2:172.21.15.78:3300/0 conn(0x7f66b00d2360 0x7f66b00d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b00b4d60 con 0x7f66b015e700 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f66ac001810 con 0x7f66b015e700 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5a23640 1 --2- 172.21.15.46:0/915609339 >> v2:172.21.15.78:3300/0 conn(0x7f66b00d2360 0x7f66b00d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f66b00d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 >> v1:172.21.15.46:6789/0 conn(0x7f66b015e700 legacy=0x7f66b00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f669c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c003ef0 0x7f669c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f669c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00b53b0 con 0x7f669c004d40 2024-04-15T15:17:56.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00b4a80 con 0x7f669c003ef0 2024-04-15T15:17:56.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f669c00b9c0 con 0x7f66b015eae0 2024-04-15T15:17:56.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f669c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6224640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f669c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b5a23640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c003ef0 0x7f669c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 -- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 msgr2=0x7f669c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b015eae0 0x7f669c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 -- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c003ef0 msgr2=0x7f669c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f669c003ef0 0x7f669c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.958+0000 7f66b6a25640 1 -- 172.21.15.46:0/915609339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669c003520 con 0x7f669c004d40 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.959+0000 7f66b6a25640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f669c0051c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f66a800e350 tx=0x7f669c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9560b3cfa9f98d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.959+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8039070 con 0x7f669c004d40 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.959+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a802cd70 con 0x7f669c004d40 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.959+0000 7f66b5222640 1 -- 172.21.15.46:0/915609339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8009de0 con 0x7f669c004d40 2024-04-15T15:17:56.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b8488640 1 -- 172.21.15.46:0/915609339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 msgr2=0x7f669c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b8488640 1 --2- 172.21.15.46:0/915609339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f669c0051c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f66a800e350 tx=0x7f669c005830 comp rx=0 tx=0).stop 2024-04-15T15:17:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b6a25640 1 -- 172.21.15.46:0/915609339 reap_dead start 2024-04-15T15:17:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b8488640 1 -- 172.21.15.46:0/915609339 shutdown_connections 2024-04-15T15:17:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b8488640 1 -- 172.21.15.46:0/915609339 >> 172.21.15.46:0/915609339 conn(0x7f66b008bf60 msgr2=0x7f66b00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b8488640 1 -- 172.21.15.46:0/915609339 shutdown_connections 2024-04-15T15:17:56.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.960+0000 7f66b8488640 1 -- 172.21.15.46:0/915609339 wait complete. 2024-04-15T15:17:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.961+0000 7f66b8488640 1 Processor -- start 2024-04-15T15:17:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.961+0000 7f66b8488640 1 -- start start 2024-04-15T15:17:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b8488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f66b0153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b8488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b0160460 0x7f66b0161990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b8488640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00b9b10 0x7f66b00b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b8488640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00d1d40 con 0x7f669c004d40 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b8488640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00b4450 con 0x7f66b00b9b10 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b8488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f66b00d1ec0 con 0x7f66b0160460 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f66b0153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f66b0153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42034/0 (socket says 172.21.15.46:42034) 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 -- 172.21.15.46:0/618943150 learned_addr learned my addr 172.21.15.46:0/618943150 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b5a23640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b0160460 0x7f66b0161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6a25640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00b9b10 0x7f66b00b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 -- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b0160460 msgr2=0x7f66b0161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b0160460 0x7f66b0161990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 -- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00b9b10 msgr2=0x7f66b00b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00b9b10 0x7f66b00b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.962+0000 7f66b6224640 1 -- 172.21.15.46:0/618943150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b01608f0 con 0x7f669c004d40 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.963+0000 7f66b6224640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f66b0153e30 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00f910 tx=0x7f66ac00fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.963+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac0072e0 con 0x7f669c004d40 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.963+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66ac018070 con 0x7f669c004d40 2024-04-15T15:17:56.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.963+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b0160bd0 con 0x7f669c004d40 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.963+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac00d7f0 con 0x7f669c004d40 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.963+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b0161190 con 0x7f669c004d40 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.964+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b00689e0 con 0x7f669c004d40 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.964+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f66ac00d990 con 0x7f669c004d40 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.965+0000 7f66a6ffd640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f669406c9b0 0x7f669406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.965+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f66ac08bb50 con 0x7f669c004d40 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.965+0000 7f66b5a23640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f669406c9b0 0x7f669406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:56.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.965+0000 7f66b5a23640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f669406c9b0 0x7f669406ee70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f66b00ce210 tx=0x7f66a0009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:56.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:56.969+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f66ac0599d0 con 0x7f669c004d40 2024-04-15T15:17:57.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.158+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f66b00d53a0 con 0x7f669406c9b0 2024-04-15T15:17:57.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.159+0000 7f66a6ffd640 1 -- 172.21.15.46:0/618943150 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f66b00d53a0 con 0x7f669406c9b0 2024-04-15T15:17:57.161 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f669406c9b0 msgr2=0x7f669406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f669406c9b0 0x7f669406ee70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f66b00ce210 tx=0x7f66a0009190 comp rx=0 tx=0).stop 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 msgr2=0x7f66b0153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f66b0153e30 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00f910 tx=0x7f66ac00fdd0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 shutdown_connections 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f669406c9b0 0x7f669406ee70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66b00b9b10 0x7f66b00b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f66b0160460 0x7f66b0161990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 --2- 172.21.15.46:0/618943150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f669c004d40 0x7f66b0153e30 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 >> 172.21.15.46:0/618943150 conn(0x7f66b008bf60 msgr2=0x7f66b00d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 shutdown_connections 2024-04-15T15:17:57.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.161+0000 7f66b8488640 1 -- 172.21.15.46:0/618943150 wait complete. 2024-04-15T15:17:57.173 INFO:tasks.ceph:Scrubbing osd.3 2024-04-15T15:17:57.174 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:57.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.281+0000 7f4f96a33640 1 Processor -- start 2024-04-15T15:17:57.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.281+0000 7f4f96a33640 1 -- start start 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f96a33640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f90162df0 0x7f4f900b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f96a33640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f900b1a20 con 0x7f4f9015eb70 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f96a33640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f900d0fe0 con 0x7f4f900b1520 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f96a33640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4f900d0d90 con 0x7f4f90162df0 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f94fd0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4f90162df0 0x7f4f900b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f8ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4f9015eb70 0x7f4f9009b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59348/0 (socket says 172.21.15.46:59348) 2024-04-15T15:17:57.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f8ffff640 1 -- 172.21.15.46:0/926192574 learned_addr learned my addr 172.21.15.46:0/926192574 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f94fd0640 1 -- 172.21.15.46:0/926192574 >> v1:172.21.15.78:6789/0 conn(0x7f4f900b1520 legacy=0x7f4f901626e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f94fd0640 1 -- 172.21.15.46:0/926192574 >> v1:172.21.15.46:6789/0 conn(0x7f4f9015eb70 legacy=0x7f4f9009b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f94fd0640 1 -- 172.21.15.46:0/926192574 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f900d12c0 con 0x7f4f90162df0 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.282+0000 7f4f8effd640 1 -- 172.21.15.46:0/926192574 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3958795955 0 0) 0x7f4f900b1a20 con 0x7f4f9015eb70 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.283+0000 7f4f94fd0640 1 --2- 172.21.15.46:0/926192574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b4a90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d90 tx=0x7f4f8802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51074a23a70cf279 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.283+0000 7f4f8effd640 1 -- 172.21.15.46:0/926192574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88039070 con 0x7f4f90162df0 2024-04-15T15:17:57.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.283+0000 7f4f8effd640 1 -- 172.21.15.46:0/926192574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f8802cc60 con 0x7f4f90162df0 2024-04-15T15:17:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.283+0000 7f4f8effd640 1 -- 172.21.15.46:0/926192574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88034660 con 0x7f4f90162df0 2024-04-15T15:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 -- 172.21.15.46:0/926192574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 msgr2=0x7f4f900b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 --2- 172.21.15.46:0/926192574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b4a90 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d90 tx=0x7f4f8802beb0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 -- 172.21.15.46:0/926192574 shutdown_connections 2024-04-15T15:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 --2- 172.21.15.46:0/926192574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b4a90 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 -- 172.21.15.46:0/926192574 >> 172.21.15.46:0/926192574 conn(0x7f4f900b3f70 msgr2=0x7f4f900b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 -- 172.21.15.46:0/926192574 shutdown_connections 2024-04-15T15:17:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.284+0000 7f4f96a33640 1 -- 172.21.15.46:0/926192574 wait complete. 2024-04-15T15:17:57.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.285+0000 7f4f96a33640 1 Processor -- start 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.285+0000 7f4f96a33640 1 -- start start 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f96a33640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f900b1520 0x7f4f900cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f96a33640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f9015eb70 0x7f4f9015d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f96a33640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f96a33640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9006d160 con 0x7f4f90162df0 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f96a33640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9015ff60 con 0x7f4f9015eb70 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f96a33640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9006d2e0 con 0x7f4f900b1520 2024-04-15T15:17:57.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42062/0 (socket says 172.21.15.46:42062) 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 -- 172.21.15.46:0/1628405116 learned_addr learned my addr 172.21.15.46:0/1628405116 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f8ffff640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f900b1520 0x7f4f900cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f8f7fe640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f9015eb70 0x7f4f9015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 -- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f900b1520 msgr2=0x7f4f900cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4f900b1520 0x7f4f900cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 -- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f9015eb70 msgr2=0x7f4f9015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f9015eb70 0x7f4f9015d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 -- 172.21.15.46:0/1628405116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9015e0a0 con 0x7f4f90162df0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.286+0000 7f4f94fd0640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b2f10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d90 tx=0x7f4f88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.287+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88039050 con 0x7f4f90162df0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.287+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f8803a040 con 0x7f4f90162df0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.287+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f900c61c0 con 0x7f4f90162df0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.287+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8803e610 con 0x7f4f90162df0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.287+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f900c6780 con 0x7f4f90162df0 2024-04-15T15:17:57.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.287+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f900689e0 con 0x7f4f90162df0 2024-04-15T15:17:57.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.288+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4f88047020 con 0x7f4f90162df0 2024-04-15T15:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8cff9640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f7c06c980 0x7f4f7c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f4f880adc60 con 0x7f4f90162df0 2024-04-15T15:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8ffff640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f7c06c980 0x7f4f7c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8cff9640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f4f7c072ec0 0x7f4f7c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 --> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f88039e50 con 0x7f4f7c072ec0 2024-04-15T15:17:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f4f880b03b0 con 0x7f4f90162df0 2024-04-15T15:17:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8f7fe640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f4f7c072ec0 0x7f4f7c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8ffff640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f7c06c980 0x7f4f7c06ee40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4f80007f70 tx=0x7f4f80007f00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.289+0000 7f4f8f7fe640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f4f7c072ec0 0x7f4f7c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.290+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== osd.3 v2:172.21.15.46:6808/3958614464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f88039e50 con 0x7f4f7c072ec0 2024-04-15T15:17:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.303+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 --> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4f900b8400 con 0x7f4f7c072ec0 2024-04-15T15:17:57.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.303+0000 7f4f8cff9640 1 -- 172.21.15.46:0/1628405116 <== osd.3 v2:172.21.15.46:6808/3958614464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4f900b8400 con 0x7f4f7c072ec0 2024-04-15T15:17:57.305 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:57.306 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:57.306 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f4f7c072ec0 msgr2=0x7f4f7c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:6808/3958614464,v1:172.21.15.46:6809/3958614464] conn(0x7f4f7c072ec0 0x7f4f7c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f7c06c980 msgr2=0x7f4f7c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4f7c06c980 0x7f4f7c06ee40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4f80007f70 tx=0x7f4f80007f00 comp rx=0 tx=0).stop 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 msgr2=0x7f4f900b2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 --2- 172.21.15.46:0/1628405116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f90162df0 0x7f4f900b2f10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f4f88009d90 tx=0x7f4f88004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:57.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f94fd0640 1 -- 172.21.15.46:0/1628405116 reap_dead start 2024-04-15T15:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 shutdown_connections 2024-04-15T15:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 >> 172.21.15.46:0/1628405116 conn(0x7f4f900b3f70 msgr2=0x7f4f900cd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 shutdown_connections 2024-04-15T15:17:57.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.304+0000 7f4f96a33640 1 -- 172.21.15.46:0/1628405116 wait complete. 2024-04-15T15:17:57.316 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-15T15:17:57.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.419+0000 7f71cdae6640 1 Processor -- start 2024-04-15T15:17:57.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71cdae6640 1 -- start start 2024-04-15T15:17:57.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71cdae6640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c809ac40 0x7f71c80d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71cdae6640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c80b49d0 0x7f71c80ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71cdae6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71c80d1210 con 0x7f71c80ce570 2024-04-15T15:17:57.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71cdae6640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f71c80b1b70 con 0x7f71c809ac40 2024-04-15T15:17:57.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71cdae6640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80d14b0 con 0x7f71c80b49d0 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c809ac40 0x7f71c80d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71c77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71c809ac40 0x7f71c80d0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42072/0 (socket says 172.21.15.46:42072) 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.420+0000 7f71c77fe640 1 -- 172.21.15.46:0/2929768637 learned_addr learned my addr 172.21.15.46:0/2929768637 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.421+0000 7f71c6ffd640 1 --2- 172.21.15.46:0/2929768637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c80b49d0 0x7f71c80ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.421+0000 7f71c77fe640 1 -- 172.21.15.46:0/2929768637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c80b49d0 msgr2=0x7f71c80ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.421+0000 7f71c77fe640 1 --2- 172.21.15.46:0/2929768637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c80b49d0 0x7f71c80ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.421+0000 7f71c77fe640 1 -- 172.21.15.46:0/2929768637 >> v1:172.21.15.78:6789/0 conn(0x7f71c80ce570 legacy=0x7f71c80b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.421+0000 7f71c77fe640 1 -- 172.21.15.46:0/2929768637 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c80cd550 con 0x7f71c809ac40 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71c77fe640 1 --2- 172.21.15.46:0/2929768637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c809ac40 0x7f71c80d0cd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009870 tx=0x7f71b802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47617331801ef0a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71c67fc640 1 -- 172.21.15.46:0/2929768637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b802f840 con 0x7f71c809ac40 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71c67fc640 1 -- 172.21.15.46:0/2929768637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71b802f9e0 con 0x7f71c809ac40 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71c67fc640 1 -- 172.21.15.46:0/2929768637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b802fdb0 con 0x7f71c809ac40 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 -- 172.21.15.46:0/2929768637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c809ac40 msgr2=0x7f71c80d0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 --2- 172.21.15.46:0/2929768637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c809ac40 0x7f71c80d0cd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f71b8009870 tx=0x7f71b802ba20 comp rx=0 tx=0).stop 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 -- 172.21.15.46:0/2929768637 shutdown_connections 2024-04-15T15:17:57.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 --2- 172.21.15.46:0/2929768637 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c80b49d0 0x7f71c80ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 --2- 172.21.15.46:0/2929768637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c809ac40 0x7f71c80d0cd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 -- 172.21.15.46:0/2929768637 >> 172.21.15.46:0/2929768637 conn(0x7f71c80b3eb0 msgr2=0x7f71c80b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 -- 172.21.15.46:0/2929768637 shutdown_connections 2024-04-15T15:17:57.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.422+0000 7f71cdae6640 1 -- 172.21.15.46:0/2929768637 wait complete. 2024-04-15T15:17:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.423+0000 7f71cdae6640 1 Processor -- start 2024-04-15T15:17:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 -- start start 2024-04-15T15:17:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c809ac40 0x7f71c80c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71c80b49d0 0x7f71c806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 0x7f71c80c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f71c80b2430 con 0x7f71c80ce570 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f71c80cd1e0 con 0x7f71c80b49d0 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71cdae6640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f71c80b25b0 con 0x7f71c809ac40 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 0x7f71c80c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71c7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 0x7f71c80c6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42088/0 (socket says 172.21.15.46:42088) 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.424+0000 7f71c7fff640 1 -- 172.21.15.46:0/3399914400 learned_addr learned my addr 172.21.15.46:0/3399914400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c77fe640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c809ac40 0x7f71c80c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c6ffd640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71c80b49d0 0x7f71c806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c7fff640 1 -- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c809ac40 msgr2=0x7f71c80c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c7fff640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c809ac40 0x7f71c80c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c7fff640 1 -- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71c80b49d0 msgr2=0x7f71c806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c7fff640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71c80b49d0 0x7f71c806caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c7fff640 1 -- 172.21.15.46:0/3399914400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c806d260 con 0x7f71c80ce570 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c77fe640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c809ac40 0x7f71c80c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c6ffd640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71c80b49d0 0x7f71c806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71c7fff640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 0x7f71c80c6160 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c6b0 tx=0x7f71c000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0003b70 con 0x7f71c80ce570 2024-04-15T15:17:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c0003d30 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c806d540 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c0005310 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.425+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c806db00 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.426+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71c80690f0 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.427+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f71c00054b0 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.427+0000 7f71affff640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f71b406c980 0x7f71b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.427+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f71c008b8c0 con 0x7f71c80ce570 2024-04-15T15:17:57.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.427+0000 7f71c77fe640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f71b406c980 0x7f71b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.428+0000 7f71c77fe640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f71b406c980 0x7f71b406ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f71b8036000 tx=0x7f71b8039040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.431+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71c80690f0 con 0x7f71c80ce570 2024-04-15T15:17:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.615+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f71c80689e0 con 0x7f71b406c980 2024-04-15T15:17:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.616+0000 7f71affff640 1 -- 172.21.15.46:0/3399914400 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f71c80689e0 con 0x7f71b406c980 2024-04-15T15:17:57.617 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 3 to deep-scrub 2024-04-15T15:17:57.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f71b406c980 msgr2=0x7f71b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f71b406c980 0x7f71b406ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f71b8036000 tx=0x7f71b8039040 comp rx=0 tx=0).stop 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 msgr2=0x7f71c80c6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 0x7f71c80c6160 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f71c000c6b0 tx=0x7f71c000cb70 comp rx=0 tx=0).stop 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 shutdown_connections 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f71b406c980 0x7f71b406ee40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71c80ce570 0x7f71c80c6160 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71c80b49d0 0x7f71c806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 --2- 172.21.15.46:0/3399914400 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f71c809ac40 0x7f71c80c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.617+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 >> 172.21.15.46:0/3399914400 conn(0x7f71c80b3eb0 msgr2=0x7f71c80af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.618+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 shutdown_connections 2024-04-15T15:17:57.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.618+0000 7f71cdae6640 1 -- 172.21.15.46:0/3399914400 wait complete. 2024-04-15T15:17:57.630 INFO:tasks.ceph:Scrubbing osd.4 2024-04-15T15:17:57.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:57.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.738+0000 7f0bff89a640 1 Processor -- start 2024-04-15T15:17:57.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.738+0000 7f0bff89a640 1 -- start start 2024-04-15T15:17:57.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bff89a640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f0bf815eb70 0x7f0bf80d39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bff89a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0bf80cfe80 0x7f0bf806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bff89a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf80d2c00 con 0x7f0bf80d1070 2024-04-15T15:17:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bff89a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf80d2ee0 con 0x7f0bf80cfe80 2024-04-15T15:17:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bff89a640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf80d31c0 con 0x7f0bf815eb70 2024-04-15T15:17:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0bf80cfe80 0x7f0bf806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0bf80cfe80 0x7f0bf806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42104/0 (socket says 172.21.15.46:42104) 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 -- 172.21.15.46:0/3730795641 learned_addr learned my addr 172.21.15.46:0/3730795641 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfd636640 1 --2- 172.21.15.46:0/3730795641 >> v2:172.21.15.78:3300/0 conn(0x7f0bf815eb70 0x7f0bf80d39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 -- 172.21.15.46:0/3730795641 >> v2:172.21.15.78:3300/0 conn(0x7f0bf815eb70 msgr2=0x7f0bf80d39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 --2- 172.21.15.46:0/3730795641 >> v2:172.21.15.78:3300/0 conn(0x7f0bf815eb70 0x7f0bf80d39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 -- 172.21.15.46:0/3730795641 >> v1:172.21.15.78:6789/0 conn(0x7f0bf80d1070 legacy=0x7f0bf80b7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.739+0000 7f0bfce35640 1 -- 172.21.15.46:0/3730795641 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf80d34a0 con 0x7f0bf80cfe80 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.740+0000 7f0bfce35640 1 --2- 172.21.15.46:0/3730795641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf80cfe80 0x7f0bf806d0b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0bf808cf00 tx=0x7f0be802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd52bf2391be8264 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.740+0000 7f0beffff640 1 -- 172.21.15.46:0/3730795641 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8035020 con 0x7f0bf80cfe80 2024-04-15T15:17:57.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.740+0000 7f0beffff640 1 -- 172.21.15.46:0/3730795641 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be8031870 con 0x7f0bf80cfe80 2024-04-15T15:17:57.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.740+0000 7f0beffff640 1 -- 172.21.15.46:0/3730795641 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8031c40 con 0x7f0bf80cfe80 2024-04-15T15:17:57.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 -- 172.21.15.46:0/3730795641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf80cfe80 msgr2=0x7f0bf806d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 --2- 172.21.15.46:0/3730795641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf80cfe80 0x7f0bf806d0b0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0bf808cf00 tx=0x7f0be802b910 comp rx=0 tx=0).stop 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 -- 172.21.15.46:0/3730795641 shutdown_connections 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 --2- 172.21.15.46:0/3730795641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf80cfe80 0x7f0bf806d0b0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 --2- 172.21.15.46:0/3730795641 >> v2:172.21.15.78:3300/0 conn(0x7f0bf815eb70 0x7f0bf80d39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 -- 172.21.15.46:0/3730795641 >> 172.21.15.46:0/3730795641 conn(0x7f0bf80b61d0 msgr2=0x7f0bf80b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 -- 172.21.15.46:0/3730795641 shutdown_connections 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.741+0000 7f0bff89a640 1 -- 172.21.15.46:0/3730795641 wait complete. 2024-04-15T15:17:57.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.742+0000 7f0bff89a640 1 Processor -- start 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.742+0000 7f0bff89a640 1 -- start start 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf80cfe80 0x7f0bf8161800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0bf80d1070 0x7f0bf80c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf815eb70 0x7f0bf80ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf806d770 con 0x7f0bf815eb70 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf806d5f0 con 0x7f0bf80cfe80 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf80d14e0 con 0x7f0bf80d1070 2024-04-15T15:17:57.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf815eb70 0x7f0bf80ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf815eb70 0x7f0bf80ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42112/0 (socket says 172.21.15.46:42112) 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 -- 172.21.15.46:0/2254797649 learned_addr learned my addr 172.21.15.46:0/2254797649 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfd636640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf80cfe80 0x7f0bf8161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfce35640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0bf80d1070 0x7f0bf80c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 -- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0bf80d1070 msgr2=0x7f0bf80c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f0bf80d1070 0x7f0bf80c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 -- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf80cfe80 msgr2=0x7f0bf8161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0bf80cfe80 0x7f0bf8161800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 -- 172.21.15.46:0/2254797649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf81492d0 con 0x7f0bf815eb70 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bfde37640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf815eb70 0x7f0bf80ca4b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c410 tx=0x7f0bf000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0011790 con 0x7f0bf815eb70 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf0017070 con 0x7f0bf815eb70 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0011bb0 con 0x7f0bf815eb70 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.743+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf81495b0 con 0x7f0bf815eb70 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.744+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf8149b70 con 0x7f0bf815eb70 2024-04-15T15:17:57.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.744+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0bf8149f50 con 0x7f0bf815eb70 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.745+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0bf001c460 con 0x7f0bf815eb70 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.745+0000 7f0bedffb640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0bdc06c980 0x7f0bdc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.745+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f0bf008b650 con 0x7f0bf815eb70 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.745+0000 7f0bedffb640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7f0bdc072ec0 0x7f0bdc075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.745+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 --> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bf001fec0 con 0x7f0bdc072ec0 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.745+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f0bf0092050 con 0x7f0bf815eb70 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.746+0000 7f0bfce35640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7f0bdc072ec0 0x7f0bdc075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.746+0000 7f0bfd636640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0bdc06c980 0x7f0bdc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.746+0000 7f0bfce35640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7f0bdc072ec0 0x7f0bdc075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.746+0000 7f0bfd636640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0bdc06c980 0x7f0bdc06ee40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40099c0 tx=0x7f0bf4003990 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.747+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== osd.4 v2:172.21.15.78:6800/3745610613 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bf001fec0 con 0x7f0bdc072ec0 2024-04-15T15:17:57.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.760+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 --> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0bf80b5fc0 con 0x7f0bdc072ec0 2024-04-15T15:17:57.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.760+0000 7f0bedffb640 1 -- 172.21.15.46:0/2254797649 <== osd.4 v2:172.21.15.78:6800/3745610613 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0bf80b5fc0 con 0x7f0bdc072ec0 2024-04-15T15:17:57.762 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:57.763 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:57.763 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7f0bdc072ec0 msgr2=0x7f0bdc075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6800/3745610613,v1:172.21.15.78:6801/3745610613] conn(0x7f0bdc072ec0 0x7f0bdc075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0bdc06c980 msgr2=0x7f0bdc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f0bdc06c980 0x7f0bdc06ee40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40099c0 tx=0x7f0bf4003990 comp rx=0 tx=0).stop 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf815eb70 msgr2=0x7f0bf80ca4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 --2- 172.21.15.46:0/2254797649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bf815eb70 0x7f0bf80ca4b0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c410 tx=0x7f0bf000c8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bfde37640 1 -- 172.21.15.46:0/2254797649 reap_dead start 2024-04-15T15:17:57.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 shutdown_connections 2024-04-15T15:17:57.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 >> 172.21.15.46:0/2254797649 conn(0x7f0bf80b61d0 msgr2=0x7f0bf80b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 shutdown_connections 2024-04-15T15:17:57.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.761+0000 7f0bff89a640 1 -- 172.21.15.46:0/2254797649 wait complete. 2024-04-15T15:17:57.774 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-15T15:17:57.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.878+0000 7f4e4ba4a640 1 Processor -- start 2024-04-15T15:17:57.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.878+0000 7f4e4ba4a640 1 -- start start 2024-04-15T15:17:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e4ba4a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e440d5150 0x7f4e440d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e4ba4a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e4415eb70 con 0x7f4e4415e790 2024-04-15T15:17:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e4ba4a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e440d3c80 con 0x7f4e440d15e0 2024-04-15T15:17:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e4ba4a640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f4e440d3790 con 0x7f4e440d5150 2024-04-15T15:17:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e497e6640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4e4415e790 0x7f4e440b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59352/0 (socket says 172.21.15.46:59352) 2024-04-15T15:17:57.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e49fe7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4e440d5150 0x7f4e440d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e497e6640 1 -- 172.21.15.46:0/3386459391 learned_addr learned my addr 172.21.15.46:0/3386459391 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e49fe7640 1 -- 172.21.15.46:0/3386459391 >> v1:172.21.15.78:6789/0 conn(0x7f4e440d15e0 legacy=0x7f4e440d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e49fe7640 1 -- 172.21.15.46:0/3386459391 >> v1:172.21.15.46:6789/0 conn(0x7f4e4415e790 legacy=0x7f4e440b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e43fff640 1 -- 172.21.15.46:0/3386459391 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1924319283 0 0) 0x7f4e4415eb70 con 0x7f4e4415e790 2024-04-15T15:17:57.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.879+0000 7f4e49fe7640 1 -- 172.21.15.46:0/3386459391 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e440d3f60 con 0x7f4e440d5150 2024-04-15T15:17:57.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.880+0000 7f4e49fe7640 1 --2- 172.21.15.46:0/3386459391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440d5150 0x7f4e440d1a90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009d90 tx=0x7f4e3402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810f5747281adf73 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.880+0000 7f4e43fff640 1 -- 172.21.15.46:0/3386459391 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34039070 con 0x7f4e440d5150 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.880+0000 7f4e43fff640 1 -- 172.21.15.46:0/3386459391 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e3402cbd0 con 0x7f4e440d5150 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.880+0000 7f4e43fff640 1 -- 172.21.15.46:0/3386459391 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34034440 con 0x7f4e440d5150 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/3386459391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440d5150 msgr2=0x7f4e440d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/3386459391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440d5150 0x7f4e440d1a90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009d90 tx=0x7f4e3402be20 comp rx=0 tx=0).stop 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/3386459391 shutdown_connections 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/3386459391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e440d5150 0x7f4e440d1a90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/3386459391 >> 172.21.15.46:0/3386459391 conn(0x7f4e44067b40 msgr2=0x7f4e44067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:57.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/3386459391 shutdown_connections 2024-04-15T15:17:57.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.881+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/3386459391 wait complete. 2024-04-15T15:17:57.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.882+0000 7f4e4ba4a640 1 Processor -- start 2024-04-15T15:17:57.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.882+0000 7f4e4ba4a640 1 -- start start 2024-04-15T15:17:57.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4e440d15e0 0x7f4e440c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e440d5150 0x7f4e440b99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 0x7f4e440b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e440b7db0 con 0x7f4e4415e790 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f4e440d45b0 con 0x7f4e440d5150 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f4e440d29c0 con 0x7f4e440d15e0 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 0x7f4e440b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 0x7f4e440b9f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42132/0 (socket says 172.21.15.46:42132) 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 -- 172.21.15.46:0/4086384734 learned_addr learned my addr 172.21.15.46:0/4086384734 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e497e6640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4e440d15e0 0x7f4e440c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e48fe5640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e440d5150 0x7f4e440b99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 -- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4e440d15e0 msgr2=0x7f4e440c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4e440d15e0 0x7f4e440c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 -- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e440d5150 msgr2=0x7f4e440b99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e440d5150 0x7f4e440b99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 -- 172.21.15.46:0/4086384734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e440c9460 con 0x7f4e4415e790 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e49fe7640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 0x7f4e440b9f20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4e34037040 tx=0x7f4e34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34039050 con 0x7f4e4415e790 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e34045070 con 0x7f4e4415e790 2024-04-15T15:17:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e440c9740 con 0x7f4e4415e790 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.883+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3403e850 con 0x7f4e4415e790 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.884+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e440c9d00 con 0x7f4e4415e790 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.885+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e440689e0 con 0x7f4e4415e790 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.885+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4e3403e9f0 con 0x7f4e4415e790 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.885+0000 7f4e41ffb640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4e3006c980 0x7f4e3006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.886+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f4e340ae880 con 0x7f4e4415e790 2024-04-15T15:17:57.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.886+0000 7f4e497e6640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4e3006c980 0x7f4e3006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:57.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.886+0000 7f4e497e6640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4e3006c980 0x7f4e3006ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c000f60 tx=0x7f4e3c007d00 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:57.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:57.890+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e34032e20 con 0x7f4e4415e790 2024-04-15T15:17:58.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.072+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f4e440b76a0 con 0x7f4e3006c980 2024-04-15T15:17:58.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.073+0000 7f4e41ffb640 1 -- 172.21.15.46:0/4086384734 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4e440b76a0 con 0x7f4e3006c980 2024-04-15T15:17:58.075 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 4 to deep-scrub 2024-04-15T15:17:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4e3006c980 msgr2=0x7f4e3006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4e3006c980 0x7f4e3006ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c000f60 tx=0x7f4e3c007d00 comp rx=0 tx=0).stop 2024-04-15T15:17:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 msgr2=0x7f4e440b9f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 0x7f4e440b9f20 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4e34037040 tx=0x7f4e34004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:58.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 shutdown_connections 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f4e3006c980 0x7f4e3006ee40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e4415e790 0x7f4e440b9f20 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e440d5150 0x7f4e440b99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 --2- 172.21.15.46:0/4086384734 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f4e440d15e0 0x7f4e440c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 >> 172.21.15.46:0/4086384734 conn(0x7f4e44067b40 msgr2=0x7f4e44074df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 shutdown_connections 2024-04-15T15:17:58.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.075+0000 7f4e4ba4a640 1 -- 172.21.15.46:0/4086384734 wait complete. 2024-04-15T15:17:58.087 INFO:tasks.ceph:Scrubbing osd.5 2024-04-15T15:17:58.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.194+0000 7fb81b750640 1 Processor -- start 2024-04-15T15:17:58.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 -- start start 2024-04-15T15:17:58.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fb81415e7f0 0x7fb81409b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb81409a9d0 0x7fb81409ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fb81409b380 con 0x7fb81409a9d0 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fb8140cd0f0 con 0x7fb81415e7f0 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.195+0000 7fb81b750640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb81406ced0 con 0x7fb8140d05e0 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb81409a9d0 0x7fb81409ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb81409a9d0 0x7fb81409ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42134/0 (socket says 172.21.15.46:42134) 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 -- 172.21.15.46:0/3570382677 learned_addr learned my addr 172.21.15.46:0/3570382677 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb819ced640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb8194ec640 1 --2- 172.21.15.46:0/3570382677 >> v2:172.21.15.78:3300/0 conn(0x7fb81415e7f0 0x7fb81409b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 -- 172.21.15.46:0/3570382677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 msgr2=0x7fb8140d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 -- 172.21.15.46:0/3570382677 >> v2:172.21.15.78:3300/0 conn(0x7fb81415e7f0 msgr2=0x7fb81409b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 --2- 172.21.15.46:0/3570382677 >> v2:172.21.15.78:3300/0 conn(0x7fb81415e7f0 0x7fb81409b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb818ceb640 1 -- 172.21.15.46:0/3570382677 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81406d1b0 con 0x7fb81409a9d0 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb8194ec640 1 --2- 172.21.15.46:0/3570382677 >> v2:172.21.15.78:3300/0 conn(0x7fb81415e7f0 0x7fb81409b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.196+0000 7fb819ced640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:58.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb818ceb640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81409a9d0 0x7fb81409ae40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb814072820 tx=0x7fb80802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce75f9b70ce6382a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb813fff640 1 -- 172.21.15.46:0/3570382677 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808034070 con 0x7fb81409a9d0 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb813fff640 1 -- 172.21.15.46:0/3570382677 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb808030860 con 0x7fb81409a9d0 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb813fff640 1 -- 172.21.15.46:0/3570382677 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808030c30 con 0x7fb81409a9d0 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 -- 172.21.15.46:0/3570382677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81409a9d0 msgr2=0x7fb81409ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81409a9d0 0x7fb81409ae40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb814072820 tx=0x7fb80802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 -- 172.21.15.46:0/3570382677 shutdown_connections 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81409a9d0 0x7fb81409ae40 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 --2- 172.21.15.46:0/3570382677 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 --2- 172.21.15.46:0/3570382677 >> v2:172.21.15.78:3300/0 conn(0x7fb81415e7f0 0x7fb81409b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.197+0000 7fb81b750640 1 -- 172.21.15.46:0/3570382677 >> 172.21.15.46:0/3570382677 conn(0x7fb8140b3f40 msgr2=0x7fb8140b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.198+0000 7fb81b750640 1 -- 172.21.15.46:0/3570382677 shutdown_connections 2024-04-15T15:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.198+0000 7fb81b750640 1 -- 172.21.15.46:0/3570382677 wait complete. 2024-04-15T15:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 Processor -- start 2024-04-15T15:17:58.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 -- start start 2024-04-15T15:17:58.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81409a9d0 0x7fb814149410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81415e7f0 0x7fb8140c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb81415ebd0 con 0x7fb81415e7f0 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fb8140ccd80 con 0x7fb81409a9d0 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.199+0000 7fb81b750640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fb81406caf0 con 0x7fb8140d05e0 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81415e7f0 0x7fb8140c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81415e7f0 0x7fb8140c64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42144/0 (socket says 172.21.15.46:42144) 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 -- 172.21.15.46:0/1007418268 learned_addr learned my addr 172.21.15.46:0/1007418268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb818ceb640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb8194ec640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81409a9d0 0x7fb814149410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 -- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 msgr2=0x7fb8140c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fb8140d05e0 0x7fb8140c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 -- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81409a9d0 msgr2=0x7fb814149410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb81409a9d0 0x7fb814149410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 -- 172.21.15.46:0/1007418268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8140c8c20 con 0x7fb81415e7f0 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb819ced640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81415e7f0 0x7fb8140c64a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00c7c0 tx=0x7fb80c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c015020 con 0x7fb81415e7f0 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb80c004230 con 0x7fb81415e7f0 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8140c8f00 con 0x7fb81415e7f0 2024-04-15T15:17:58.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb80c007620 con 0x7fb81415e7f0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.200+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8140ca610 con 0x7fb81415e7f0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.201+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb81414cad0 con 0x7fb81415e7f0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.202+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb80c018020 con 0x7fb81415e7f0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.202+0000 7fb811ffb640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb80006c980 0x7fb80006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fb80c08c110 con 0x7fb81415e7f0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb811ffb640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb800072ec0 0x7fb800075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 --> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8000759d0 con 0x7fb800072ec0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb80c08c5f0 con 0x7fb81415e7f0 2024-04-15T15:17:58.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb8194ec640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb80006c980 0x7fb80006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb818ceb640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb800072ec0 0x7fb800075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb8194ec640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb80006c980 0x7fb80006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb80400d540 tx=0x7fb804007810 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.203+0000 7fb818ceb640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb800072ec0 0x7fb800075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.204+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== osd.5 v2:172.21.15.78:6816/2916087727 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb8000759d0 con 0x7fb800072ec0 2024-04-15T15:17:58.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.217+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 --> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb814158ed0 con 0x7fb800072ec0 2024-04-15T15:17:58.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.217+0000 7fb811ffb640 1 -- 172.21.15.46:0/1007418268 <== osd.5 v2:172.21.15.78:6816/2916087727 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb8080309d0 con 0x7fb800072ec0 2024-04-15T15:17:58.219 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:58.220 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:58.220 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb800072ec0 msgr2=0x7fb800075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6816/2916087727,v1:172.21.15.78:6818/2916087727] conn(0x7fb800072ec0 0x7fb800075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb80006c980 msgr2=0x7fb80006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fb80006c980 0x7fb80006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb80400d540 tx=0x7fb804007810 comp rx=0 tx=0).stop 2024-04-15T15:17:58.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81415e7f0 msgr2=0x7fb8140c64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 --2- 172.21.15.46:0/1007418268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb81415e7f0 0x7fb8140c64a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb80c00c7c0 tx=0x7fb80c00cc80 comp rx=0 tx=0).stop 2024-04-15T15:17:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb819ced640 1 -- 172.21.15.46:0/1007418268 reap_dead start 2024-04-15T15:17:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 shutdown_connections 2024-04-15T15:17:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 >> 172.21.15.46:0/1007418268 conn(0x7fb8140b3f40 msgr2=0x7fb81415c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 shutdown_connections 2024-04-15T15:17:58.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.218+0000 7fb81b750640 1 -- 172.21.15.46:0/1007418268 wait complete. 2024-04-15T15:17:58.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-15T15:17:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.333+0000 7fc5afec1640 1 Processor -- start 2024-04-15T15:17:58.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.334+0000 7fc5afec1640 1 -- start start 2024-04-15T15:17:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.334+0000 7fc5afec1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.334+0000 7fc5afec1640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5a80d2360 0x7fc5a80d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.334+0000 7fc5afec1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a80b47a0 con 0x7fc5a815eae0 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.334+0000 7fc5afec1640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7fc5a80b53b0 con 0x7fc5a80d2360 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.334+0000 7fc5afec1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a80b4a80 con 0x7fc5a815e700 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5a80d2360 0x7fc5a80d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc5a80d2360 0x7fc5a80d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42158/0 (socket says 172.21.15.46:42158) 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 -- 172.21.15.46:0/1671262612 learned_addr learned my addr 172.21.15.46:0/1671262612 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5adc5d640 1 --2- 172.21.15.46:0/1671262612 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 -- 172.21.15.46:0/1671262612 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 msgr2=0x7fc5a80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 --2- 172.21.15.46:0/1671262612 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 -- 172.21.15.46:0/1671262612 >> v1:172.21.15.78:6789/0 conn(0x7fc5a815eae0 legacy=0x7fc5a80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.335+0000 7fc5ad45c640 1 -- 172.21.15.46:0/1671262612 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a80b4d60 con 0x7fc5a80d2360 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5ad45c640 1 --2- 172.21.15.46:0/1671262612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a80d2b60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc59c009870 tx=0x7fc59c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87b200db501ed01 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5acc5b640 1 -- 172.21.15.46:0/1671262612 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c035020 con 0x7fc5a80d2360 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5acc5b640 1 -- 172.21.15.46:0/1671262612 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc59c031640 con 0x7fc5a80d2360 2024-04-15T15:17:58.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5acc5b640 1 -- 172.21.15.46:0/1671262612 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59c02d040 con 0x7fc5a80d2360 2024-04-15T15:17:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5afec1640 1 -- 172.21.15.46:0/1671262612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 msgr2=0x7fc5a80d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5afec1640 1 --2- 172.21.15.46:0/1671262612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a80d2b60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc59c009870 tx=0x7fc59c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5afec1640 1 -- 172.21.15.46:0/1671262612 shutdown_connections 2024-04-15T15:17:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5afec1640 1 --2- 172.21.15.46:0/1671262612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a80d2b60 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5afec1640 1 --2- 172.21.15.46:0/1671262612 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.336+0000 7fc5afec1640 1 -- 172.21.15.46:0/1671262612 >> 172.21.15.46:0/1671262612 conn(0x7fc5a808bf60 msgr2=0x7fc5a80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.337+0000 7fc5afec1640 1 -- 172.21.15.46:0/1671262612 shutdown_connections 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.337+0000 7fc5afec1640 1 -- 172.21.15.46:0/1671262612 wait complete. 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 Processor -- start 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 -- start start 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a814caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a815eae0 0x7fc5a80b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a80b5d80 con 0x7fc5a80d2360 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a80b4450 con 0x7fc5a815eae0 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5afec1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a80d1d40 con 0x7fc5a815e700 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.338+0000 7fc5adc5d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a814caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a814caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42174/0 (socket says 172.21.15.46:42174) 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 -- 172.21.15.46:0/2408724750 learned_addr learned my addr 172.21.15.46:0/2408724750 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5ad45c640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5ae45e640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a815eae0 0x7fc5a80b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 -- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 msgr2=0x7fc5a80c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 -- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a815eae0 msgr2=0x7fc5a80b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a815eae0 0x7fc5a80b7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 -- 172.21.15.46:0/2408724750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a8160230 con 0x7fc5a80d2360 2024-04-15T15:17:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5adc5d640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a814caa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc598007fa0 tx=0x7fc59800e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598014620 con 0x7fc5a80d2360 2024-04-15T15:17:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc598019070 con 0x7fc5a80d2360 2024-04-15T15:17:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598014a00 con 0x7fc5a80d2360 2024-04-15T15:17:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a8160510 con 0x7fc5a80d2360 2024-04-15T15:17:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.339+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a8160ad0 con 0x7fc5a80d2360 2024-04-15T15:17:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.340+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a80689e0 con 0x7fc5a80d2360 2024-04-15T15:17:58.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.341+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc598014ba0 con 0x7fc5a80d2360 2024-04-15T15:17:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.341+0000 7fc5a67fc640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc59406c980 0x7fc59406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.341+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fc59808c350 con 0x7fc5a80d2360 2024-04-15T15:17:58.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.341+0000 7fc5ad45c640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc59406c980 0x7fc59406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.342+0000 7fc5ad45c640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc59406c980 0x7fc59406ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc59c036000 tx=0x7fc59c008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.345+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc59805a940 con 0x7fc5a80d2360 2024-04-15T15:17:58.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.530+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fc5a815e000 con 0x7fc59406c980 2024-04-15T15:17:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.531+0000 7fc5a67fc640 1 -- 172.21.15.46:0/2408724750 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5a815e000 con 0x7fc59406c980 2024-04-15T15:17:58.533 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 5 to deep-scrub 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc59406c980 msgr2=0x7fc59406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc59406c980 0x7fc59406ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc59c036000 tx=0x7fc59c008030 comp rx=0 tx=0).stop 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 msgr2=0x7fc5a814caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a814caa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc598007fa0 tx=0x7fc59800e570 comp rx=0 tx=0).stop 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 shutdown_connections 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc59406c980 0x7fc59406ee40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5a815eae0 0x7fc5a80b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc5a815e700 0x7fc5a80c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 --2- 172.21.15.46:0/2408724750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5a80d2360 0x7fc5a814caa0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 >> 172.21.15.46:0/2408724750 conn(0x7fc5a808bf60 msgr2=0x7fc5a80d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 shutdown_connections 2024-04-15T15:17:58.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.533+0000 7fc5afec1640 1 -- 172.21.15.46:0/2408724750 wait complete. 2024-04-15T15:17:58.546 INFO:tasks.ceph:Scrubbing osd.6 2024-04-15T15:17:58.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.654+0000 7f25fd7a8640 1 Processor -- start 2024-04-15T15:17:58.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.654+0000 7f25fd7a8640 1 -- start start 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25fd7a8640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f25f815fe10 0x7f25f80b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25fd7a8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25f80d0ec0 0x7f25f809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25fd7a8640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25f81601f0 con 0x7f25f80b2f80 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25fd7a8640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f25f80b1ec0 con 0x7f25f80d0ec0 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25fd7a8640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f25f80b21a0 con 0x7f25f815fe10 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25ee7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f25f80b2f80 0x7f25f80d07b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59366/0 (socket says 172.21.15.46:59366) 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f25f80d0ec0 0x7f25f809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25ee7fc640 1 -- 172.21.15.46:0/1185628220 learned_addr learned my addr 172.21.15.46:0/1185628220 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f6ffd640 1 --2- 172.21.15.46:0/1185628220 >> v2:172.21.15.78:3300/0 conn(0x7f25f815fe10 0x7f25f80b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f77fe640 1 -- 172.21.15.46:0/1185628220 >> v2:172.21.15.78:3300/0 conn(0x7f25f815fe10 msgr2=0x7f25f80b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f77fe640 1 --2- 172.21.15.46:0/1185628220 >> v2:172.21.15.78:3300/0 conn(0x7f25f815fe10 0x7f25f80b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f77fe640 1 -- 172.21.15.46:0/1185628220 >> v1:172.21.15.46:6789/0 conn(0x7f25f80b2f80 legacy=0x7f25f80d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f77fe640 1 -- 172.21.15.46:0/1185628220 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f80b2480 con 0x7f25f80d0ec0 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f67fc640 1 -- 172.21.15.46:0/1185628220 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059272603 0 0) 0x7f25f81601f0 con 0x7f25f80b2f80 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.655+0000 7f25f6ffd640 1 --2- 172.21.15.46:0/1185628220 >> v2:172.21.15.78:3300/0 conn(0x7f25f815fe10 0x7f25f80b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.656+0000 7f25f77fe640 1 --2- 172.21.15.46:0/1185628220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f809a9d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f25f80734f0 tx=0x7f25f002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661017283c81da7f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.656+0000 7f25f67fc640 1 -- 172.21.15.46:0/1185628220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0039070 con 0x7f25f80d0ec0 2024-04-15T15:17:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.656+0000 7f25f67fc640 1 -- 172.21.15.46:0/1185628220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f002cbf0 con 0x7f25f80d0ec0 2024-04-15T15:17:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.656+0000 7f25f67fc640 1 -- 172.21.15.46:0/1185628220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f0034440 con 0x7f25f80d0ec0 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.656+0000 7f25fd7a8640 1 -- 172.21.15.46:0/1185628220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 msgr2=0x7f25f809a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.656+0000 7f25fd7a8640 1 --2- 172.21.15.46:0/1185628220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f809a9d0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f25f80734f0 tx=0x7f25f002be40 comp rx=0 tx=0).stop 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.657+0000 7f25fd7a8640 1 -- 172.21.15.46:0/1185628220 shutdown_connections 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.657+0000 7f25fd7a8640 1 --2- 172.21.15.46:0/1185628220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f809a9d0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.657+0000 7f25fd7a8640 1 --2- 172.21.15.46:0/1185628220 >> v2:172.21.15.78:3300/0 conn(0x7f25f815fe10 0x7f25f80b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.657+0000 7f25fd7a8640 1 -- 172.21.15.46:0/1185628220 >> 172.21.15.46:0/1185628220 conn(0x7f25f80b3f70 msgr2=0x7f25f80b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.657+0000 7f25fd7a8640 1 -- 172.21.15.46:0/1185628220 shutdown_connections 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.657+0000 7f25fd7a8640 1 -- 172.21.15.46:0/1185628220 wait complete. 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.658+0000 7f25fd7a8640 1 Processor -- start 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.658+0000 7f25fd7a8640 1 -- start start 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25f80b2f80 0x7f25f81f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f81f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25f815fe10 0x7f25f80cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25f809af10 con 0x7f25f80d0ec0 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f25f80b2a40 con 0x7f25f815fe10 2024-04-15T15:17:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f25f809b090 con 0x7f25f80b2f80 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f81f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f81f1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42188/0 (socket says 172.21.15.46:42188) 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 -- 172.21.15.46:0/3576447390 learned_addr learned my addr 172.21.15.46:0/3576447390 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25f77fe640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25f815fe10 0x7f25f80cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25f6ffd640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25f80b2f80 0x7f25f81f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 -- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25f80b2f80 msgr2=0x7f25f81f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f25f80b2f80 0x7f25f81f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 -- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25f815fe10 msgr2=0x7f25f80cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25f815fe10 0x7f25f80cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 -- 172.21.15.46:0/3576447390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f81f64b0 con 0x7f25f80d0ec0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25ee7fc640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f81f1790 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c4d0 tx=0x7f25e000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0018070 con 0x7f25f80d0ec0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25e0007500 con 0x7f25f80d0ec0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f81487a0 con 0x7f25f80d0ec0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.659+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e0004a20 con 0x7f25f80d0ec0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.660+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f8148d60 con 0x7f25f80d0ec0 2024-04-15T15:17:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.660+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f25f81f20d0 con 0x7f25f80d0ec0 2024-04-15T15:17:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.661+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f25e001b070 con 0x7f25f80d0ec0 2024-04-15T15:17:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.661+0000 7f25effff640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25d806c980 0x7f25d806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f25e008acd0 con 0x7f25f80d0ec0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25effff640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7f25d8072ec0 0x7f25d8075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 --> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25d80759d0 con 0x7f25d8072ec0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f25e008d420 con 0x7f25f80d0ec0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25f6ffd640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25d806c980 0x7f25d806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25f77fe640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7f25d8072ec0 0x7f25d8075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25f6ffd640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25d806c980 0x7f25d806ee40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f25e800aac0 tx=0x7f25e8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.662+0000 7f25f77fe640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7f25d8072ec0 0x7f25d8075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.663+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== osd.6 v2:172.21.15.78:6817/3189427077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f25d80759d0 con 0x7f25d8072ec0 2024-04-15T15:17:58.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.676+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 --> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f25f80689e0 con 0x7f25d8072ec0 2024-04-15T15:17:58.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25effff640 1 -- 172.21.15.46:0/3576447390 <== osd.6 v2:172.21.15.78:6817/3189427077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f25f80689e0 con 0x7f25d8072ec0 2024-04-15T15:17:58.679 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:58.679 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:58.679 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:58.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7f25d8072ec0 msgr2=0x7f25d8075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6817/3189427077,v1:172.21.15.78:6819/3189427077] conn(0x7f25d8072ec0 0x7f25d8075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25d806c980 msgr2=0x7f25d806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f25d806c980 0x7f25d806ee40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f25e800aac0 tx=0x7f25e8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:58.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 msgr2=0x7f25f81f1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 --2- 172.21.15.46:0/3576447390 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25f80d0ec0 0x7f25f81f1790 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c4d0 tx=0x7f25e000c990 comp rx=0 tx=0).stop 2024-04-15T15:17:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25f77fe640 1 -- 172.21.15.46:0/3576447390 reap_dead start 2024-04-15T15:17:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 shutdown_connections 2024-04-15T15:17:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.677+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 >> 172.21.15.46:0/3576447390 conn(0x7f25f80b3f70 msgr2=0x7f25f815bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.678+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 shutdown_connections 2024-04-15T15:17:58.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.678+0000 7f25fd7a8640 1 -- 172.21.15.46:0/3576447390 wait complete. 2024-04-15T15:17:58.690 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-15T15:17:58.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.794+0000 7ff9cd104640 1 Processor -- start 2024-04-15T15:17:58.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.794+0000 7ff9cd104640 1 -- start start 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9cd104640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9c8070940 0x7ff9c809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9cd104640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7ff9c8160310 0x7ff9c8070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9cd104640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9c80d05e0 con 0x7ff9c80d0da0 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9cd104640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c8071260 con 0x7ff9c8070940 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9cd104640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7ff9c815f720 con 0x7ff9c8160310 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9c8070940 0x7ff9c809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff9c8070940 0x7ff9c809b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42192/0 (socket says 172.21.15.46:42192) 2024-04-15T15:17:58.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 -- 172.21.15.46:0/3632763435 learned_addr learned my addr 172.21.15.46:0/3632763435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c659c640 1 --2- 172.21.15.46:0/3632763435 >> v2:172.21.15.78:3300/0 conn(0x7ff9c8160310 0x7ff9c8070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 -- 172.21.15.46:0/3632763435 >> v2:172.21.15.78:3300/0 conn(0x7ff9c8160310 msgr2=0x7ff9c8070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 --2- 172.21.15.46:0/3632763435 >> v2:172.21.15.78:3300/0 conn(0x7ff9c8160310 0x7ff9c8070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 -- 172.21.15.46:0/3632763435 >> v1:172.21.15.78:6789/0 conn(0x7ff9c80d0da0 legacy=0x7ff9c80d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.795+0000 7ff9c6d9d640 1 -- 172.21.15.46:0/3632763435 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c815fa00 con 0x7ff9c8070940 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.796+0000 7ff9c6d9d640 1 --2- 172.21.15.46:0/3632763435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8070940 0x7ff9c809b880 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80734f0 tx=0x7ff9b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbd97899d59cc0d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.796+0000 7ff9c5d9b640 1 -- 172.21.15.46:0/3632763435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b802ca80 con 0x7ff9c8070940 2024-04-15T15:17:58.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.796+0000 7ff9c5d9b640 1 -- 172.21.15.46:0/3632763435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b802cc20 con 0x7ff9c8070940 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.796+0000 7ff9c5d9b640 1 -- 172.21.15.46:0/3632763435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b80343f0 con 0x7ff9c8070940 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 -- 172.21.15.46:0/3632763435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8070940 msgr2=0x7ff9c809b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3632763435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8070940 0x7ff9c809b880 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff9c80734f0 tx=0x7ff9b802bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 -- 172.21.15.46:0/3632763435 shutdown_connections 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3632763435 >> v2:172.21.15.78:3300/0 conn(0x7ff9c8160310 0x7ff9c8070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3632763435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8070940 0x7ff9c809b880 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 -- 172.21.15.46:0/3632763435 >> 172.21.15.46:0/3632763435 conn(0x7ff9c80b3f40 msgr2=0x7ff9c80b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 -- 172.21.15.46:0/3632763435 shutdown_connections 2024-04-15T15:17:58.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.797+0000 7ff9cd104640 1 -- 172.21.15.46:0/3632763435 wait complete. 2024-04-15T15:17:58.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.798+0000 7ff9cd104640 1 Processor -- start 2024-04-15T15:17:58.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.798+0000 7ff9cd104640 1 -- start start 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9cd104640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9c8070940 0x7ff9c80b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9cd104640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff9c80d0da0 0x7ff9c814fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9cd104640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 0x7ff9c814ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9cd104640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c80d18e0 con 0x7ff9c8160310 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9cd104640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7ff9c80d0880 con 0x7ff9c8070940 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9cd104640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7ff9c8160770 con 0x7ff9c80d0da0 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 0x7ff9c814ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 0x7ff9c814ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42194/0 (socket says 172.21.15.46:42194) 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 -- 172.21.15.46:0/3734665972 learned_addr learned my addr 172.21.15.46:0/3734665972 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c6d9d640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9c8070940 0x7ff9c80b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 -- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff9c80d0da0 msgr2=0x7ff9c814fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c659c640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff9c80d0da0 0x7ff9c814fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff9c80d0da0 0x7ff9c814fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 -- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9c8070940 msgr2=0x7ff9c80b76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9c8070940 0x7ff9c80b76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c759e640 1 -- 172.21.15.46:0/3734665972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c80a0390 con 0x7ff9c8160310 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c6d9d640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9c8070940 0x7ff9c80b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.799+0000 7ff9c659c640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff9c80d0da0 0x7ff9c814fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.800+0000 7ff9c759e640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 0x7ff9c814ffb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b490 tx=0x7ff9c000b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.800+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0015020 con 0x7ff9c8160310 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.800+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c0011850 con 0x7ff9c8160310 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.800+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9c815daa0 con 0x7ff9c8160310 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.800+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0011c40 con 0x7ff9c8160310 2024-04-15T15:17:58.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.800+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9c80c5f40 con 0x7ff9c8160310 2024-04-15T15:17:58.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.801+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9c80689e0 con 0x7ff9c8160310 2024-04-15T15:17:58.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.801+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff9c0018020 con 0x7ff9c8160310 2024-04-15T15:17:58.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.802+0000 7ff9bf7fe640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff9a806c980 0x7ff9a806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:58.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.802+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7ff9c008b900 con 0x7ff9c8160310 2024-04-15T15:17:58.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.802+0000 7ff9c6d9d640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff9a806c980 0x7ff9a806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:58.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.803+0000 7ff9c6d9d640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff9a806c980 0x7ff9a806ee40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8035000 tx=0x7ff9b8030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:58.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.806+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9c001f020 con 0x7ff9c8160310 2024-04-15T15:17:58.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.991+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7ff9c80b4ca0 con 0x7ff9a806c980 2024-04-15T15:17:58.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.992+0000 7ff9bf7fe640 1 -- 172.21.15.46:0/3734665972 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff9c80b4ca0 con 0x7ff9a806c980 2024-04-15T15:17:58.994 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 6 to deep-scrub 2024-04-15T15:17:58.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff9a806c980 msgr2=0x7ff9a806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff9a806c980 0x7ff9a806ee40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8035000 tx=0x7ff9b8030000 comp rx=0 tx=0).stop 2024-04-15T15:17:58.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 msgr2=0x7ff9c814ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:58.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 0x7ff9c814ffb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b490 tx=0x7ff9c000b950 comp rx=0 tx=0).stop 2024-04-15T15:17:58.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 shutdown_connections 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7ff9a806c980 0x7ff9a806ee40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff9c8160310 0x7ff9c814ffb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7ff9c80d0da0 0x7ff9c814fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 --2- 172.21.15.46:0/3734665972 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9c8070940 0x7ff9c80b76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 >> 172.21.15.46:0/3734665972 conn(0x7ff9c80b3f40 msgr2=0x7ff9c80b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 shutdown_connections 2024-04-15T15:17:58.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:58.994+0000 7ff9cd104640 1 -- 172.21.15.46:0/3734665972 wait complete. 2024-04-15T15:17:59.007 INFO:tasks.ceph:Scrubbing osd.7 2024-04-15T15:17:59.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:59.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.114+0000 7f462e182640 1 Processor -- start 2024-04-15T15:17:59.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.114+0000 7f462e182640 1 -- start start 2024-04-15T15:17:59.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f462e182640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280cd5f0 0x7f46280ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f462e182640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46280d1010 con 0x7f46280b21b0 2024-04-15T15:17:59.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f462e182640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f462806c1a0 con 0x7f46280d0c30 2024-04-15T15:17:59.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f462e182640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f46280ce0b0 con 0x7f46280cd5f0 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f4627fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f46280b21b0 0x7f46280b0b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59370/0 (socket says 172.21.15.46:59370) 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f4627fff640 1 -- 172.21.15.46:0/947318721 learned_addr learned my addr 172.21.15.46:0/947318721 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f4626ffd640 1 --2- 172.21.15.46:0/947318721 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280cd5f0 0x7f46280ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184472151 0 0) 0x7f46280d1010 con 0x7f46280b21b0 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.115+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4610003520 con 0x7f46280b21b0 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.116+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056797318 0 0) 0x7f462806c1a0 con 0x7f46280d0c30 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.116+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46280d1010 con 0x7f46280d0c30 2024-04-15T15:17:59.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.116+0000 7f4626ffd640 1 -- 172.21.15.46:0/947318721 >> v1:172.21.15.78:6789/0 conn(0x7f46280d0c30 legacy=0x7f462806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.116+0000 7f4626ffd640 1 -- 172.21.15.46:0/947318721 >> v1:172.21.15.46:6789/0 conn(0x7f46280b21b0 legacy=0x7f46280b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.116+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 921908551 0 0) 0x7f4610003520 con 0x7f46280b21b0 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.116+0000 7f4626ffd640 1 -- 172.21.15.46:0/947318721 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462806c480 con 0x7f46280cd5f0 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f4626ffd640 1 --2- 172.21.15.46:0/947318721 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280cd5f0 0x7f46280ce540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f461c0097d0 tx=0x7f461c02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873f6a3c8bb4015 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c037070 con 0x7f46280cd5f0 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f461c030510 con 0x7f46280cd5f0 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f46267fc640 1 -- 172.21.15.46:0/947318721 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46200019f0 con 0x7f46280cd5f0 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 -- 172.21.15.46:0/947318721 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280cd5f0 msgr2=0x7f46280ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 --2- 172.21.15.46:0/947318721 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280cd5f0 0x7f46280ce540 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f461c0097d0 tx=0x7f461c02b800 comp rx=0 tx=0).stop 2024-04-15T15:17:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 -- 172.21.15.46:0/947318721 shutdown_connections 2024-04-15T15:17:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 --2- 172.21.15.46:0/947318721 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280cd5f0 0x7f46280ce540 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 -- 172.21.15.46:0/947318721 >> 172.21.15.46:0/947318721 conn(0x7f46280b2e70 msgr2=0x7f46280b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 -- 172.21.15.46:0/947318721 shutdown_connections 2024-04-15T15:17:59.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.117+0000 7f462e182640 1 -- 172.21.15.46:0/947318721 wait complete. 2024-04-15T15:17:59.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.118+0000 7f462e182640 1 Processor -- start 2024-04-15T15:17:59.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 -- start start 2024-04-15T15:17:59.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280b21b0 0x7f4628154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46280cd5f0 0x7f4628159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46280d0c30 0x7f4628159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f46280b1220 con 0x7f46280cd5f0 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f46280cea80 con 0x7f46280d0c30 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f462e182640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f46280b13a0 con 0x7f46280b21b0 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46280cd5f0 0x7f4628159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f4626ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46280cd5f0 0x7f4628159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42208/0 (socket says 172.21.15.46:42208) 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.119+0000 7f4626ffd640 1 -- 172.21.15.46:0/1341243512 learned_addr learned my addr 172.21.15.46:0/1341243512 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:59.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f46277fe640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280b21b0 0x7f4628154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4627fff640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46280d0c30 0x7f4628159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4626ffd640 1 -- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280b21b0 msgr2=0x7f4628154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4626ffd640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f46280b21b0 0x7f4628154de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4626ffd640 1 -- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46280d0c30 msgr2=0x7f4628159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4626ffd640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46280d0c30 0x7f4628159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4626ffd640 1 -- 172.21.15.46:0/1341243512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f462815ad50 con 0x7f46280cd5f0 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f4626ffd640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46280cd5f0 0x7f4628159350 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f461c02b840 tx=0x7f461c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c041020 con 0x7f46280cd5f0 2024-04-15T15:17:59.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f461c03c870 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46280c65a0 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461c037040 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.120+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f462815a0d0 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.121+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f46280690f0 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.122+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f461c044020 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.122+0000 7f460ffff640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f461406c980 0x7f461406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.122+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f461c0ad5e0 con 0x7f46280cd5f0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.122+0000 7f460ffff640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f4614072ec0 0x7f4614075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.122+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 --> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f461c008eb0 con 0x7f4614072ec0 2024-04-15T15:17:59.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.122+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f461c0b4050 con 0x7f46280cd5f0 2024-04-15T15:17:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.123+0000 7f46277fe640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f461406c980 0x7f461406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.123+0000 7f4627fff640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f4614072ec0 0x7f4614075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.123+0000 7f46277fe640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f461406c980 0x7f461406ee40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f46180011e0 tx=0x7f4618009040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.123+0000 7f4627fff640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f4614072ec0 0x7f4614075300 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.124+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== osd.7 v2:172.21.15.78:6808/563779653 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f461c008eb0 con 0x7f4614072ec0 2024-04-15T15:17:59.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.137+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 --> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4628075a80 con 0x7f4614072ec0 2024-04-15T15:17:59.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f460ffff640 1 -- 172.21.15.46:0/1341243512 <== osd.7 v2:172.21.15.78:6808/563779653 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4628075a80 con 0x7f4614072ec0 2024-04-15T15:17:59.140 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-15T15:17:59.140 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:59.140 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-15T15:17:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f4614072ec0 msgr2=0x7f4614075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6808/563779653,v1:172.21.15.78:6809/563779653] conn(0x7f4614072ec0 0x7f4614075300 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f461406c980 msgr2=0x7f461406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f461406c980 0x7f461406ee40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f46180011e0 tx=0x7f4618009040 comp rx=0 tx=0).stop 2024-04-15T15:17:59.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46280cd5f0 msgr2=0x7f4628159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 --2- 172.21.15.46:0/1341243512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46280cd5f0 0x7f4628159350 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f461c02b840 tx=0x7f461c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f4627fff640 1 -- 172.21.15.46:0/1341243512 reap_dead start 2024-04-15T15:17:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 shutdown_connections 2024-04-15T15:17:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 >> 172.21.15.46:0/1341243512 conn(0x7f46280b2e70 msgr2=0x7f462806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 shutdown_connections 2024-04-15T15:17:59.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.138+0000 7f462e182640 1 -- 172.21.15.46:0/1341243512 wait complete. 2024-04-15T15:17:59.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-15T15:17:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.254+0000 7f35c6352640 1 Processor -- start 2024-04-15T15:17:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.254+0000 7f35c6352640 1 -- start start 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35c6352640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f35c00b5230 0x7f35c0072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35c6352640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c0073080 con 0x7f35c0158770 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35c6352640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c00b4190 con 0x7f35c0072c60 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35c6352640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f35c00b4da0 con 0x7f35c00b5230 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f35c0158770 0x7f35c015bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:59380/0 (socket says 172.21.15.46:59380) 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bffff640 1 -- 172.21.15.46:0/206273997 learned_addr learned my addr 172.21.15.46:0/206273997 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914736867 0 0) 0x7f35c0073080 con 0x7f35c0158770 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35b7fff640 1 --2- 172.21.15.46:0/206273997 >> v2:172.21.15.78:3300/0 conn(0x7f35c00b5230 0x7f35c0072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35ac003440 con 0x7f35c0158770 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3580100481 0 0) 0x7f35ac003440 con 0x7f35c0158770 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 >> v2:172.21.15.78:3300/0 conn(0x7f35c00b5230 msgr2=0x7f35c0072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 --2- 172.21.15.46:0/206273997 >> v2:172.21.15.78:3300/0 conn(0x7f35c00b5230 0x7f35c0072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 >> v1:172.21.15.78:6789/0 conn(0x7f35c0072c60 legacy=0x7f35c006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c00b4470 con 0x7f35c0158770 2024-04-15T15:17:59.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.255+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7f35b0001670 con 0x7f35c0158770 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 >> v1:172.21.15.46:6789/0 conn(0x7f35c0158770 legacy=0x7f35c015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35ac003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35ac008850 0x7f35ac00acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35ac00b4c0 0x7f35ac00d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35c00b4da0 con 0x7f35ac003aa0 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35ac00e260 con 0x7f35ac00b4c0 2024-04-15T15:17:59.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f35ac00e460 con 0x7f35ac008850 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bffff640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35ac003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35c48ef640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35ac00b4c0 0x7f35ac00d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35b7fff640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35ac008850 0x7f35ac00acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bffff640 1 -- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35ac008850 msgr2=0x7f35ac00acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bffff640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35ac008850 0x7f35ac00acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bffff640 1 -- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35ac00b4c0 msgr2=0x7f35ac00d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bffff640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35ac00b4c0 0x7f35ac00d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35bffff640 1 -- 172.21.15.46:0/206273997 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ac003440 con 0x7f35ac003aa0 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35c48ef640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35ac00b4c0 0x7f35ac00d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.256+0000 7f35c48ef640 1 -- 172.21.15.46:0/206273997 reap_dead start 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35bffff640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35ac003ec0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c240 tx=0x7f35ac003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabece863ed9a530 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0007cf0 con 0x7f35ac003aa0 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b0030070 con 0x7f35ac003aa0 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35bf7fe640 1 -- 172.21.15.46:0/206273997 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0005d30 con 0x7f35ac003aa0 2024-04-15T15:17:59.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 -- 172.21.15.46:0/206273997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 msgr2=0x7f35ac003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35ac003ec0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c240 tx=0x7f35ac003730 comp rx=0 tx=0).stop 2024-04-15T15:17:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 -- 172.21.15.46:0/206273997 shutdown_connections 2024-04-15T15:17:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 --2- 172.21.15.46:0/206273997 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35ac003ec0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 -- 172.21.15.46:0/206273997 >> 172.21.15.46:0/206273997 conn(0x7f35c006be00 msgr2=0x7f35c006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 -- 172.21.15.46:0/206273997 shutdown_connections 2024-04-15T15:17:59.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.257+0000 7f35c6352640 1 -- 172.21.15.46:0/206273997 wait complete. 2024-04-15T15:17:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 Processor -- start 2024-04-15T15:17:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 -- start start 2024-04-15T15:17:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35c0155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35c015e790 0x7f35c015ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35c00ced70 0x7f35c00cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35c0071fa0 con 0x7f35ac003aa0 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35c00b4a30 con 0x7f35c015e790 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35c6352640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f35c0158b50 con 0x7f35c00ced70 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35c0155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.259+0000 7f35bffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35c0155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42212/0 (socket says 172.21.15.46:42212) 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 -- 172.21.15.46:0/2867003756 learned_addr learned my addr 172.21.15.46:0/2867003756 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35b7fff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35c015e790 0x7f35c015ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35c48ef640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35c00ced70 0x7f35c00cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 -- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35c00ced70 msgr2=0x7f35c00cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35c00ced70 0x7f35c00cf1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 -- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35c015e790 msgr2=0x7f35c015ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35c015e790 0x7f35c015ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 -- 172.21.15.46:0/2867003756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c01f0750 con 0x7f35ac003aa0 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35b7fff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35c015e790 0x7f35c015ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bffff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35c0155b60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c360 tx=0x7f35c00b4640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0045070 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b0004bf0 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c01f0a30 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0007cf0 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.260+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c01f0ff0 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.261+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c00689e0 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.262+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f35b0007070 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.262+0000 7f35bd7fa640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f35a006c980 0x7f35a006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.262+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f35b00acbf0 con 0x7f35ac003aa0 2024-04-15T15:17:59.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.262+0000 7f35b7fff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f35a006c980 0x7f35a006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.263+0000 7f35b7fff640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f35a006c980 0x7f35a006ee40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35c006e9d0 tx=0x7f35a8008040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.266+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35b007b260 con 0x7f35ac003aa0 2024-04-15T15:17:59.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.449+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f35c00b0e70 con 0x7f35a006c980 2024-04-15T15:17:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.450+0000 7f35bd7fa640 1 -- 172.21.15.46:0/2867003756 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f35c00b0e70 con 0x7f35a006c980 2024-04-15T15:17:59.452 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 7 to deep-scrub 2024-04-15T15:17:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f35a006c980 msgr2=0x7f35a006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f35a006c980 0x7f35a006ee40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35c006e9d0 tx=0x7f35a8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 msgr2=0x7f35c0155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35c0155b60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f35b000c360 tx=0x7f35c00b4640 comp rx=0 tx=0).stop 2024-04-15T15:17:59.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 shutdown_connections 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f35a006c980 0x7f35a006ee40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35c00ced70 0x7f35c00cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35c015e790 0x7f35c015ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 --2- 172.21.15.46:0/2867003756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ac003aa0 0x7f35c0155b60 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 >> 172.21.15.46:0/2867003756 conn(0x7f35c006be00 msgr2=0x7f35c00b1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 shutdown_connections 2024-04-15T15:17:59.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.452+0000 7f35c6352640 1 -- 172.21.15.46:0/2867003756 wait complete. 2024-04-15T15:17:59.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:59.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.567+0000 7fdc50996640 1 Processor -- start 2024-04-15T15:17:59.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.567+0000 7fdc50996640 1 -- start start 2024-04-15T15:17:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc50996640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c15e700 0x7fdc4c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc50996640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fdc4c0d3270 0x7fdc4c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc50996640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc4c0b47a0 con 0x7fdc4c0d2360 2024-04-15T15:17:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc50996640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fdc4c0b53b0 con 0x7fdc4c0d3270 2024-04-15T15:17:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc50996640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4c0b4a80 con 0x7fdc4c15e700 2024-04-15T15:17:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc49d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdc4c0d2360 0x7fdc4c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58068/0 (socket says 172.21.15.46:58068) 2024-04-15T15:17:59.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc49d9b640 1 -- 172.21.15.46:0/3127980579 learned_addr learned my addr 172.21.15.46:0/3127980579 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:17:59.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4ad9d640 1 --2- 172.21.15.46:0/3127980579 >> v2:172.21.15.78:3300/0 conn(0x7fdc4c0d3270 0x7fdc4c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.568+0000 7fdc4a59c640 1 --2- 172.21.15.46:0/3127980579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c15e700 0x7fdc4c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4959a640 1 -- 172.21.15.46:0/3127980579 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445171080 0 0) 0x7fdc4c0b47a0 con 0x7fdc4c0d2360 2024-04-15T15:17:59.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4959a640 1 -- 172.21.15.46:0/3127980579 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc38003520 con 0x7fdc4c0d2360 2024-04-15T15:17:59.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4a59c640 1 -- 172.21.15.46:0/3127980579 >> v2:172.21.15.78:3300/0 conn(0x7fdc4c0d3270 msgr2=0x7fdc4c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:59.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4a59c640 1 --2- 172.21.15.46:0/3127980579 >> v2:172.21.15.78:3300/0 conn(0x7fdc4c0d3270 0x7fdc4c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4a59c640 1 -- 172.21.15.46:0/3127980579 >> v1:172.21.15.46:6789/0 conn(0x7fdc4c0d2360 legacy=0x7fdc4c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4959a640 1 -- 172.21.15.46:0/3127980579 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2238532484 0 0) 0x7fdc38003520 con 0x7fdc4c0d2360 2024-04-15T15:17:59.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4a59c640 1 -- 172.21.15.46:0/3127980579 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c0b4d60 con 0x7fdc4c15e700 2024-04-15T15:17:59.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.569+0000 7fdc4ad9d640 1 --2- 172.21.15.46:0/3127980579 >> v2:172.21.15.78:3300/0 conn(0x7fdc4c0d3270 0x7fdc4c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:59.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.570+0000 7fdc4a59c640 1 --2- 172.21.15.46:0/3127980579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c15e700 0x7fdc4c0b5840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009790 tx=0x7fdc4402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23bec8aef5d8378d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.570+0000 7fdc4959a640 1 -- 172.21.15.46:0/3127980579 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc44037070 con 0x7fdc4c15e700 2024-04-15T15:17:59.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.570+0000 7fdc4959a640 1 -- 172.21.15.46:0/3127980579 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc44030510 con 0x7fdc4c15e700 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.570+0000 7fdc4959a640 1 -- 172.21.15.46:0/3127980579 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34001670 con 0x7fdc4c15e700 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.570+0000 7fdc50996640 1 -- 172.21.15.46:0/3127980579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c15e700 msgr2=0x7fdc4c0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.570+0000 7fdc50996640 1 --2- 172.21.15.46:0/3127980579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c15e700 0x7fdc4c0b5840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc44009790 tx=0x7fdc4402b800 comp rx=0 tx=0).stop 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.571+0000 7fdc50996640 1 -- 172.21.15.46:0/3127980579 shutdown_connections 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.571+0000 7fdc50996640 1 --2- 172.21.15.46:0/3127980579 >> v2:172.21.15.78:3300/0 conn(0x7fdc4c0d3270 0x7fdc4c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.571+0000 7fdc50996640 1 --2- 172.21.15.46:0/3127980579 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c15e700 0x7fdc4c0b5840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.571+0000 7fdc50996640 1 -- 172.21.15.46:0/3127980579 >> 172.21.15.46:0/3127980579 conn(0x7fdc4c08bf60 msgr2=0x7fdc4c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:59.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.571+0000 7fdc50996640 1 -- 172.21.15.46:0/3127980579 shutdown_connections 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.571+0000 7fdc50996640 1 -- 172.21.15.46:0/3127980579 wait complete. 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.572+0000 7fdc50996640 1 Processor -- start 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.572+0000 7fdc50996640 1 -- start start 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c0d2360 0x7fdc4c1f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0d3270 0x7fdc4c0b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 0x7fdc4c0b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c0b5d80 con 0x7fdc4c15e700 2024-04-15T15:17:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4c0b4450 con 0x7fdc4c0d3270 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4c0d1d40 con 0x7fdc4c0d2360 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 0x7fdc4c0b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 0x7fdc4c0b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57516/0 (socket says 172.21.15.46:57516) 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 -- 172.21.15.46:0/4039463455 learned_addr learned my addr 172.21.15.46:0/4039463455 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4a59c640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c0d2360 0x7fdc4c1f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc49d9b640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0d3270 0x7fdc4c0b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 -- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c0d2360 msgr2=0x7fdc4c1f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c0d2360 0x7fdc4c1f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 -- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0d3270 msgr2=0x7fdc4c0b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0d3270 0x7fdc4c0b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 -- 172.21.15.46:0/4039463455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c1f1b30 con 0x7fdc4c15e700 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc4ad9d640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 0x7fdc4c0b1630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00efc0 tx=0x7fdc3c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c018030 con 0x7fdc4c15e700 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3c00cde0 con 0x7fdc4c15e700 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4c14b710 con 0x7fdc4c15e700 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.573+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c003bb0 con 0x7fdc4c15e700 2024-04-15T15:17:59.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.574+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc4c0b9450 con 0x7fdc4c15e700 2024-04-15T15:17:59.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.574+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4c0689e0 con 0x7fdc4c15e700 2024-04-15T15:17:59.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.575+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fdc3c005d10 con 0x7fdc4c15e700 2024-04-15T15:17:59.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.575+0000 7fdc42ffd640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc2c06c980 0x7fdc2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:59.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.575+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fdc3c08c5d0 con 0x7fdc4c15e700 2024-04-15T15:17:59.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.576+0000 7fdc4a59c640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc2c06c980 0x7fdc2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:59.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.576+0000 7fdc4a59c640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc2c06c980 0x7fdc2c06ee40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc44034000 tx=0x7fdc44008030 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:59.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.580+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc3c022070 con 0x7fdc4c15e700 2024-04-15T15:17:59.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.759+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc4c0ba500 con 0x7fdc2c06c980 2024-04-15T15:17:59.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.760+0000 7fdc42ffd640 1 -- 172.21.15.46:0/4039463455 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13434 (secure 0 0 0) 0x7fdc4c0ba500 con 0x7fdc2c06c980 2024-04-15T15:17:59.762 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:17:59.762 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:17:59.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc2c06c980 msgr2=0x7fdc2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc2c06c980 0x7fdc2c06ee40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc44034000 tx=0x7fdc44008030 comp rx=0 tx=0).stop 2024-04-15T15:17:59.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 msgr2=0x7fdc4c0b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:59.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 0x7fdc4c0b1630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c00efc0 tx=0x7fdc3c009f90 comp rx=0 tx=0).stop 2024-04-15T15:17:59.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 shutdown_connections 2024-04-15T15:17:59.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fdc2c06c980 0x7fdc2c06ee40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4c15e700 0x7fdc4c0b1630 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0d3270 0x7fdc4c0b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 --2- 172.21.15.46:0/4039463455 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fdc4c0d2360 0x7fdc4c1f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 >> 172.21.15.46:0/4039463455 conn(0x7fdc4c08bf60 msgr2=0x7fdc4c0c8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 shutdown_connections 2024-04-15T15:17:59.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:17:59.762+0000 7fdc50996640 1 -- 172.21.15.46:0/4039463455 wait complete. 2024-04-15T15:17:59.774 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-04-15T15:17:58.387236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238120,"kb_used_data":5288,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736600,"statfs":{"total":773094113280,"available":772850278400,"internally_reserved":0,"allocated":5414912,"data_stored":3038400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.970191"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:17:50.750872+0000","last_change":"2024-04-15T15:15:27.558855+0000","last_active":"2024-04-15T15:17:50.750872+0000","last_peered":"2024-04-15T15:17:50.750872+0000","last_clean":"2024-04-15T15:17:50.750872+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:17:50.750872+0000","last_undegraded":"2024-04-15T15:17:50.750872+0000","last_fullsized":"2024-04-15T15:17:50.750872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:26.508462+0000","last_clean_scrub_stamp":"2024-04-15T15:15:26.508462+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:04.644736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542178,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343724,"statfs":{"total":96636764160,"available":96607973376,"internally_reserved":0,"allocated":1122304,"data_stored":822576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1122304,"data_stored":822576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:59.777 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-15T15:15:26.508462+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=15, tm_sec=26, tm_wday=0, tm_yday=106, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=17, tm_sec=54, tm_wday=0, tm_yday=106, tm_isdst=0) 2024-04-15T15:17:59.777 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-15T15:18:10.163 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:18:10.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:18:19.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:18:19.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.894+0000 7f08f4789640 1 Processor -- start 2024-04-15T15:18:19.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.895+0000 7f08f4789640 1 -- start start 2024-04-15T15:18:19.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.895+0000 7f08f4789640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08ec0d2360 0x7f08ec0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:19.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f4789640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15eae0 0x7f08ec0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:19.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f4789640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec0b47a0 con 0x7f08ec15e700 2024-04-15T15:18:19.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f4789640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 -- 0x7f08ec0b53b0 con 0x7f08ec0d2360 2024-04-15T15:18:19.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f4789640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec0b4a80 con 0x7f08ec15eae0 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08ec0d2360 0x7f08ec0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08ec0d2360 0x7f08ec0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33712/0 (socket says 172.21.15.46:33712) 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 -- 172.21.15.46:0/3304490662 learned_addr learned my addr 172.21.15.46:0/3304490662 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f2d26640 1 --2- 172.21.15.46:0/3304490662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15eae0 0x7f08ec0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 -- 172.21.15.46:0/3304490662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15eae0 msgr2=0x7f08ec0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 --2- 172.21.15.46:0/3304490662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15eae0 0x7f08ec0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 -- 172.21.15.46:0/3304490662 >> v1:172.21.15.78:6789/0 conn(0x7f08ec15e700 legacy=0x7f08ec0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:19.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.896+0000 7f08f1d24640 1 -- 172.21.15.46:0/3304490662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec0b4d60 con 0x7f08ec0d2360 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.897+0000 7f08f1d24640 1 --2- 172.21.15.46:0/3304490662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0d2b60 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009870 tx=0x7f08e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c803af9bb400db0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.897+0000 7f08f1523640 1 -- 172.21.15.46:0/3304490662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0034070 con 0x7f08ec0d2360 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.897+0000 7f08f1523640 1 -- 172.21.15.46:0/3304490662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e0030840 con 0x7f08ec0d2360 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.897+0000 7f08f1523640 1 -- 172.21.15.46:0/3304490662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0030c10 con 0x7f08ec0d2360 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.897+0000 7f08f4789640 1 -- 172.21.15.46:0/3304490662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 msgr2=0x7f08ec0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.897+0000 7f08f4789640 1 --2- 172.21.15.46:0/3304490662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0d2b60 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009870 tx=0x7f08e002b980 comp rx=0 tx=0).stop 2024-04-15T15:18:19.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.898+0000 7f08f4789640 1 -- 172.21.15.46:0/3304490662 shutdown_connections 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.898+0000 7f08f4789640 1 --2- 172.21.15.46:0/3304490662 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15eae0 0x7f08ec0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.898+0000 7f08f4789640 1 --2- 172.21.15.46:0/3304490662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0d2b60 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.898+0000 7f08f4789640 1 -- 172.21.15.46:0/3304490662 >> 172.21.15.46:0/3304490662 conn(0x7f08ec08bf60 msgr2=0x7f08ec0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.898+0000 7f08f4789640 1 -- 172.21.15.46:0/3304490662 shutdown_connections 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.898+0000 7f08f4789640 1 -- 172.21.15.46:0/3304490662 wait complete. 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.899+0000 7f08f4789640 1 Processor -- start 2024-04-15T15:18:19.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.899+0000 7f08f4789640 1 -- start start 2024-04-15T15:18:19.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:19.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15e700 0x7f08ec161710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:19.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08ec15eae0 0x7f08ec1f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:19.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0d1d40 con 0x7f08ec0d2360 2024-04-15T15:18:19.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0b4450 con 0x7f08ec15eae0 2024-04-15T15:18:19.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec0d1ec0 con 0x7f08ec15e700 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0b9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33728/0 (socket says 172.21.15.46:33728) 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 -- 172.21.15.46:0/2708628623 learned_addr learned my addr 172.21.15.46:0/2708628623 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f1d24640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15e700 0x7f08ec161710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2d26640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08ec15eae0 0x7f08ec1f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 -- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15e700 msgr2=0x7f08ec161710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15e700 0x7f08ec161710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:19.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 -- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08ec15eae0 msgr2=0x7f08ec1f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08ec15eae0 0x7f08ec1f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 -- 172.21.15.46:0/2708628623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec15ef90 con 0x7f08ec0d2360 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f1d24640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15e700 0x7f08ec161710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2d26640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08ec15eae0 0x7f08ec1f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f2525640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0b9c50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f08e400c440 tx=0x7f08e4001ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4017070 con 0x7f08ec0d2360 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e40052f0 con 0x7f08ec0d2360 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec15f270 con 0x7f08ec0d2360 2024-04-15T15:18:19.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4003660 con 0x7f08ec0d2360 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.900+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec0cbd50 con 0x7f08ec0d2360 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.901+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ec0689e0 con 0x7f08ec0d2360 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.902+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f08e401a070 con 0x7f08ec0d2360 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.902+0000 7f08eaffd640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f08d806c9b0 0x7f08d806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.903+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f08e408aef0 con 0x7f08ec0d2360 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.903+0000 7f08f1d24640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f08d806c9b0 0x7f08d806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:19.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.903+0000 7f08f1d24640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f08d806c9b0 0x7f08d806ee70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f08e0036000 tx=0x7f08e003b040 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:19.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:19.907+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08e4059350 con 0x7f08ec0d2360 2024-04-15T15:18:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.089+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 --> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08ec0ba500 con 0x7f08d806c9b0 2024-04-15T15:18:20.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.090+0000 7f08eaffd640 1 -- 172.21.15.46:0/2708628623 <== mgr.4114 v2:172.21.15.78:6832/1451944854 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7f08ec0ba500 con 0x7f08d806c9b0 2024-04-15T15:18:20.092 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:18:20.092 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-15T15:18:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f08d806c9b0 msgr2=0x7f08d806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f08d806c9b0 0x7f08d806ee70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f08e0036000 tx=0x7f08e003b040 comp rx=0 tx=0).stop 2024-04-15T15:18:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 msgr2=0x7f08ec0b9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0b9c50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f08e400c440 tx=0x7f08e4001ec0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 shutdown_connections 2024-04-15T15:18:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f08d806c9b0 0x7f08d806ee70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08ec15eae0 0x7f08ec1f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f08ec15e700 0x7f08ec161710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 --2- 172.21.15.46:0/2708628623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08ec0d2360 0x7f08ec0b9c50 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 >> 172.21.15.46:0/2708628623 conn(0x7f08ec08bf60 msgr2=0x7f08ec0d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:18:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 shutdown_connections 2024-04-15T15:18:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.092+0000 7f08f4789640 1 -- 172.21.15.46:0/2708628623 wait complete. 2024-04-15T15:18:20.105 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-04-15T15:18:18.390892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238112,"kb_used_data":5288,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736608,"statfs":{"total":773094113280,"available":772850286592,"internally_reserved":0,"allocated":5414912,"data_stored":3038400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002435"},"pg_stats":[{"pgid":"1.0","version":"45'119","reported_seq":226,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:18:00.383435+0000","last_change":"2024-04-15T15:18:00.383475+0000","last_active":"2024-04-15T15:18:00.383435+0000","last_peered":"2024-04-15T15:18:00.383435+0000","last_clean":"2024-04-15T15:18:00.383435+0000","last_became_active":"2024-04-15T15:15:27.555986+0000","last_became_peered":"2024-04-15T15:15:27.555986+0000","last_unstale":"2024-04-15T15:18:00.383435+0000","last_undegraded":"2024-04-15T15:18:00.383435+0000","last_fullsized":"2024-04-15T15:18:00.383435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'119","last_scrub_stamp":"2024-04-15T15:18:00.383335+0000","last_deep_scrub":"45'119","last_deep_scrub_stamp":"2024-04-15T15:18:00.383335+0000","last_clean_scrub_stamp":"2024-04-15T15:18:00.383335+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:24:07.848127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":14,"seq":60129542182,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542182,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":1122304,"data_stored":822576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":232208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339568,"statfs":{"total":96636764160,"available":96603717632,"internally_reserved":0,"allocated":1122304,"data_stored":822576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:18:20.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-15T15:18:20.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.209+0000 7f3589ab1640 1 Processor -- start 2024-04-15T15:18:20.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.209+0000 7f3589ab1640 1 -- start start 2024-04-15T15:18:20.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3589ab1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 0x7f3584070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3589ab1640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3584071d90 0x7f358409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3589ab1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358406ff30 con 0x7f3584071240 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3589ab1640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7f3584070820 con 0x7f3584071d90 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3589ab1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f35840cecc0 con 0x7f35840ce8e0 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3582ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3584071d90 0x7f358409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f35837fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 0x7f3584070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3582ffd640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7f3584071d90 0x7f358409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.46:52486/0 (socket says 172.21.15.46:52486) 2024-04-15T15:18:20.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3582ffd640 1 -- 172.21.15.46:0/3578555546 learned_addr learned my addr 172.21.15.46:0/3578555546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f3583fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3584071240 0x7f3584160460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.46:54648/0 (socket says 172.21.15.46:54648) 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.210+0000 7f35837fe640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 0x7f3584070cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3301/0 says I am v2:172.21.15.46:48828/0 (socket says 172.21.15.46:48828) 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f3582ffd640 1 -- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 msgr2=0x7f3584070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f3582ffd640 1 --2- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 0x7f3584070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f3582ffd640 1 -- 172.21.15.46:0/3578555546 >> v1:172.21.15.78:6789/0 conn(0x7f3584071240 legacy=0x7f3584160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f3582ffd640 1 -- 172.21.15.46:0/3578555546 --> v2:172.21.15.78:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358409b9d0 con 0x7f3584071d90 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f35827fc640 1 -- 172.21.15.46:0/3578555546 <== mon.1 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1462840624 0 0) 0x7f358406ff30 con 0x7f3584071240 2024-04-15T15:18:20.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f35837fe640 1 --2- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 0x7f3584070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.211+0000 7f3582ffd640 1 --2- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3584071d90 0x7f358409a9d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3578009870 tx=0x7f357802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abfcebfc23e0e873 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f35827fc640 1 -- 172.21.15.46:0/3578555546 <== mon.1 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578035020 con 0x7f3584071d90 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f35827fc640 1 -- 172.21.15.46:0/3578555546 <== mon.1 v2:172.21.15.78:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3578031640 con 0x7f3584071d90 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f35827fc640 1 -- 172.21.15.46:0/3578555546 <== mon.1 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357802d040 con 0x7f3584071d90 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 -- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3584071d90 msgr2=0x7f358409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 --2- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3584071d90 0x7f358409a9d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3578009870 tx=0x7f357802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 -- 172.21.15.46:0/3578555546 shutdown_connections 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 --2- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3584071d90 0x7f358409a9d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 --2- 172.21.15.46:0/3578555546 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f35840ce8e0 0x7f3584070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 -- 172.21.15.46:0/3578555546 >> 172.21.15.46:0/3578555546 conn(0x7f35840b3f40 msgr2=0x7f35840b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 -- 172.21.15.46:0/3578555546 shutdown_connections 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.212+0000 7f3589ab1640 1 -- 172.21.15.46:0/3578555546 wait complete. 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.213+0000 7f3589ab1640 1 Processor -- start 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 -- start start 2024-04-15T15:18:20.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3584071240 0x7f358415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 0x7f35840b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35840ce8e0 0x7f35840cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35840704b0 con 0x7f3584071d90 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7f35840701d0 con 0x7f35840ce8e0 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.214+0000 7f3589ab1640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7f3584071970 con 0x7f3584071240 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 0x7f35840b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 0x7f35840b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33742/0 (socket says 172.21.15.46:33742) 2024-04-15T15:18:20.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 -- 172.21.15.46:0/1224312946 learned_addr learned my addr 172.21.15.46:0/1224312946 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f35837fe640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3584071240 0x7f358415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3583fff640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35840ce8e0 0x7f35840cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 -- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3584071240 msgr2=0x7f358415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3584071240 0x7f358415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 -- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35840ce8e0 msgr2=0x7f35840cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35840ce8e0 0x7f35840cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 -- 172.21.15.46:0/1224312946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35840b67b0 con 0x7f3584071d90 2024-04-15T15:18:20.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3582ffd640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 0x7f35840b5cb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f35780090b0 tx=0x7f35780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578040070 con 0x7f3584071d90 2024-04-15T15:18:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35780399f0 con 0x7f3584071d90 2024-04-15T15:18:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578035040 con 0x7f3584071d90 2024-04-15T15:18:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.215+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35840b6a90 con 0x7f3584071d90 2024-04-15T15:18:20.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.216+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35840cb1d0 con 0x7f3584071d90 2024-04-15T15:18:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.217+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35840689e0 con 0x7f3584071d90 2024-04-15T15:18:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.217+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3578038020 con 0x7f3584071d90 2024-04-15T15:18:20.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.217+0000 7f356bfff640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f357006c980 0x7f357006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.218+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f35780ad330 con 0x7f3584071d90 2024-04-15T15:18:20.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.218+0000 7f35837fe640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f357006c980 0x7f357006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.218+0000 7f35837fe640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f357006c980 0x7f357006ee40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3574009560 tx=0x7f3574009190 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:20.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.222+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f357807b1c0 con 0x7f3584071d90 2024-04-15T15:18:20.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.406+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f35840b4ca0 con 0x7f3584071d90 2024-04-15T15:18:20.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.409+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7f357807f4c0 con 0x7f3584071d90 2024-04-15T15:18:20.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.410+0000 7f356bfff640 1 -- 172.21.15.46:0/1224312946 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3578083cf0 con 0x7f3584071d90 2024-04-15T15:18:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f357006c980 msgr2=0x7f357006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f357006c980 0x7f357006ee40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3574009560 tx=0x7f3574009190 comp rx=0 tx=0).stop 2024-04-15T15:18:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 msgr2=0x7f35840b5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 0x7f35840b5cb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f35780090b0 tx=0x7f35780046c0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 shutdown_connections 2024-04-15T15:18:20.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7f357006c980 0x7f357006ee40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35840ce8e0 0x7f35840cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3584071d90 0x7f35840b5cb0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 --2- 172.21.15.46:0/1224312946 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7f3584071240 0x7f358415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 >> 172.21.15.46:0/1224312946 conn(0x7f35840b3f40 msgr2=0x7f35840c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:18:20.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 shutdown_connections 2024-04-15T15:18:20.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.412+0000 7f3589ab1640 1 -- 172.21.15.46:0/1224312946 wait complete. 2024-04-15T15:18:20.424 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:18:20.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:18:20.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.529+0000 7fc3fc488640 1 Processor -- start 2024-04-15T15:18:20.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.529+0000 7fc3fc488640 1 -- start start 2024-04-15T15:18:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3fc488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3f40d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3fc488640 1 --2- >> v2:172.21.15.78:3300/0 conn(0x7fc3f40b3160 0x7fc3f415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3fc488640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f40d0fb0 con 0x7fc3f415ef70 2024-04-15T15:18:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3fc488640 1 -- --> v2:172.21.15.78:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f415cb50 con 0x7fc3f40b3160 2024-04-15T15:18:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3fc488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f415bfd0 con 0x7fc3f40d0690 2024-04-15T15:18:20.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3faa25640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc3f415ef70 0x7fc3f415f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:55950/0 (socket says 172.21.15.46:55950) 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3faa25640 1 -- 172.21.15.46:0/3964099593 learned_addr learned my addr 172.21.15.46:0/3964099593 (peer_addr_for_me v1:172.21.15.46:0/0) 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926515324 0 0) 0x7fc3f40d0fb0 con 0x7fc3f415ef70 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3fa224640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3f40d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/3964099593 >> v2:172.21.15.78:3300/0 conn(0x7fc3f40b3160 0x7fc3f415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3e0003520 con 0x7fc3f415ef70 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3147992643 0 0) 0x7fc3e0003520 con 0x7fc3f415ef70 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 msgr2=0x7fc3f40d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:18:20.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3f40d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 >> v2:172.21.15.78:3300/0 conn(0x7fc3f40b3160 msgr2=0x7fc3f415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 --2- 172.21.15.46:0/3964099593 >> v2:172.21.15.78:3300/0 conn(0x7fc3f40b3160 0x7fc3f415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f415c2b0 con 0x7fc3f415ef70 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2755650110 0 0) 0x7fc3ec0019f0 con 0x7fc3f415ef70 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 >> v1:172.21.15.46:6789/0 conn(0x7fc3f415ef70 legacy=0x7fc3f415f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e00039b0 0x7fc3e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e0004d40 0x7fc3e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f415cb50 con 0x7fc3e0004d40 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f415bfd0 con 0x7fc3e00039b0 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.530+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e000b9c0 con 0x7fc3f40d0690 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3faa25640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e0004d40 0x7fc3e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3fa224640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e00039b0 0x7fc3e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3f9a23640 1 -- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e00039b0 msgr2=0x7fc3e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e00039b0 0x7fc3e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3f9a23640 1 -- 172.21.15.46:0/3964099593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e0004d40 msgr2=0x7fc3e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:18:20.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e0004d40 0x7fc3e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3f9a23640 1 -- 172.21.15.46:0/3964099593 --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e0003520 con 0x7fc3f40d0690 2024-04-15T15:18:20.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.531+0000 7fc3faa25640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e0004d40 0x7fc3e00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:18:20.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3e0004800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400bfc0 tx=0x7fc3e0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47fc1680ce639314 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:20.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 <== mon.2 v2:172.21.15.78:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e403b070 con 0x7fc3f40d0690 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 <== mon.2 v2:172.21.15.78:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc3e402e890 con 0x7fc3f40d0690 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3f9222640 1 -- 172.21.15.46:0/3964099593 <== mon.2 v2:172.21.15.78:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e400bc10 con 0x7fc3f40d0690 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3fc488640 1 -- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 msgr2=0x7fc3e0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3fc488640 1 --2- 172.21.15.46:0/3964099593 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f40d0690 0x7fc3e0004800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400bfc0 tx=0x7fc3e0005830 comp rx=0 tx=0).stop 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3faa25640 1 -- 172.21.15.46:0/3964099593 reap_dead start 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3fc488640 1 -- 172.21.15.46:0/3964099593 shutdown_connections 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3fc488640 1 -- 172.21.15.46:0/3964099593 >> 172.21.15.46:0/3964099593 conn(0x7fc3f406be00 msgr2=0x7fc3f406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:18:20.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3fc488640 1 -- 172.21.15.46:0/3964099593 shutdown_connections 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.532+0000 7fc3fc488640 1 -- 172.21.15.46:0/3964099593 wait complete. 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 Processor -- start 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 -- start start 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 0x7fc3f4154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 --2- >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f4156360 0x7fc3f41567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f41e5bd0 0x7fc3f41e6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f40b4fb0 con 0x7fc3f40d0690 2024-04-15T15:18:20.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f415c870 con 0x7fc3f41e5bd0 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fc488640 1 -- --> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f415bbd0 con 0x7fc3f4156360 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fa224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 0x7fc3f4154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fa224640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 0x7fc3f4154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33752/0 (socket says 172.21.15.46:33752) 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.534+0000 7fc3fa224640 1 -- 172.21.15.46:0/1011883288 learned_addr learned my addr 172.21.15.46:0/1011883288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3faa25640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f41e5bd0 0x7fc3f41e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f4156360 0x7fc3f41567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fa224640 1 -- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f4156360 msgr2=0x7fc3f41567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fa224640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f4156360 0x7fc3f41567e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fa224640 1 -- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f41e5bd0 msgr2=0x7fc3f41e6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fa224640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f41e5bd0 0x7fc3f41e6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fa224640 1 -- 172.21.15.46:0/1011883288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f4156ed0 con 0x7fc3f40d0690 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fa224640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 0x7fc3f4154300 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0011960 tx=0x7fc3f0011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f000fa90 con 0x7fc3f40d0690 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc3f000fc30 con 0x7fc3f40d0690 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f41571b0 con 0x7fc3f40d0690 2024-04-15T15:18:20.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0014440 con 0x7fc3f40d0690 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.535+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f4157770 con 0x7fc3f40d0690 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.536+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f40689e0 con 0x7fc3f40d0690 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.536+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc3f0012040 con 0x7fc3f40d0690 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.537+0000 7fc3eaffd640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc3d806c9b0 0x7fc3d806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.537+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fc3f008c0a0 con 0x7fc3f40d0690 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.537+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc3d806c9b0 0x7fc3d806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:18:20.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.538+0000 7fc3f9a23640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc3d806c9b0 0x7fc3d806ee70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4038000 tx=0x7fc3e4030000 comp rx=0 tx=0).ready entity=mgr.4114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:18:20.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.541+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3f0019030 con 0x7fc3f40d0690 2024-04-15T15:18:20.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.764+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3f415ae40 con 0x7fc3f40d0690 2024-04-15T15:18:20.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.764+0000 7fc3eaffd640 1 -- 172.21.15.46:0/1011883288 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+3866 (secure 0 0 0) 0x7fc3f0059f20 con 0x7fc3f40d0690 2024-04-15T15:18:20.767 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-15T15:18:20.767 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4940,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/353369147","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6835","nonce":353369147},{"type":"v1","addr":"172.21.15.78:6837","nonce":353369147}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6836/3465113004","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":3465113004},{"type":"v1","addr":"172.21.15.78:6836","nonce":3465113004}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5451,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/3745232368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3745232368},{"type":"v1","addr":"172.21.15.46:6834","nonce":3745232368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":6198,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3680286510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3680286510},{"type":"v1","addr":"172.21.15.46:6835","nonce":3680286510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[]} 2024-04-15T15:18:20.767 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2024-04-15T15:18:20.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc3d806c9b0 msgr2=0x7fc3d806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc3d806c9b0 0x7fc3d806ee70 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4038000 tx=0x7fc3e4030000 comp rx=0 tx=0).stop 2024-04-15T15:18:20.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 msgr2=0x7fc3f4154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 0x7fc3f4154300 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0011960 tx=0x7fc3f0011e20 comp rx=0 tx=0).stop 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 shutdown_connections 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:6832/1451944854,v1:172.21.15.78:6833/1451944854] conn(0x7fc3d806c9b0 0x7fc3d806ee70 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3f41e5bd0 0x7fc3f41e6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] conn(0x7fc3f4156360 0x7fc3f41567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 --2- 172.21.15.46:0/1011883288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3f40d0690 0x7fc3f4154300 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 >> 172.21.15.46:0/1011883288 conn(0x7fc3f406be00 msgr2=0x7fc3f4071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:18:20.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 shutdown_connections 2024-04-15T15:18:20.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-15T15:18:20.766+0000 7fc3fc488640 1 -- 172.21.15.46:0/1011883288 wait complete. 2024-04-15T15:18:20.779 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-15T15:18:20.780 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-15T15:18:20.780 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:18:20.780 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:20.789 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:18:20.789 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:18:20.790 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:20.839 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:18:20.839 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:18:20.840 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:20.852 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:18:20.852 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:18:20.852 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:20.904 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:18:20.904 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-15T15:18:20.905 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-15T15:18:20.905 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:20.928 INFO:tasks.ceph.osd.0:Stopped 2024-04-15T15:18:20.928 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-15T15:18:20.928 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:20.995 INFO:tasks.ceph.osd.1:Stopped 2024-04-15T15:18:20.995 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-15T15:18:20.995 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.070 INFO:tasks.ceph.osd.2:Stopped 2024-04-15T15:18:21.070 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-15T15:18:21.071 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.139 INFO:tasks.ceph.osd.3:Stopped 2024-04-15T15:18:21.139 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-15T15:18:21.140 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.169 INFO:tasks.ceph.osd.4:Stopped 2024-04-15T15:18:21.170 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-15T15:18:21.170 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.240 INFO:tasks.ceph.osd.5:Stopped 2024-04-15T15:18:21.240 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-15T15:18:21.240 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.313 INFO:tasks.ceph.osd.6:Stopped 2024-04-15T15:18:21.313 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-15T15:18:21.313 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.387 INFO:tasks.ceph.osd.7:Stopped 2024-04-15T15:18:21.387 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-15T15:18:21.388 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-15T15:18:21.388 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.412 INFO:tasks.ceph.mgr.y:Stopped 2024-04-15T15:18:21.412 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-15T15:18:21.412 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.458 INFO:tasks.ceph.mgr.x:Stopped 2024-04-15T15:18:21.458 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-15T15:18:21.459 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-15T15:18:21.459 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.470 INFO:tasks.ceph.mon.a:Stopped 2024-04-15T15:18:21.470 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-15T15:18:21.470 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.511 INFO:tasks.ceph.mon.b:Stopped 2024-04-15T15:18:21.512 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-15T15:18:21.512 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:18:21.564 INFO:tasks.ceph.mon.c:Stopped 2024-04-15T15:18:21.564 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-15T15:18:21.564 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | head -n 1 2024-04-15T15:18:21.599 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:18:21.599 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-15T15:18:21.840 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:18:21.841 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-15T15:18:21.915 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:18:21.915 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-15T15:18:21.972 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:18:21.972 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-15T15:18:22.023 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:18:22.023 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-15T15:18:22.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:18:22.213 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-15T15:18:22.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:18:22.309 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-15T15:18:22.367 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:18:22.368 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-15T15:18:22.431 INFO:tasks.ceph:Archiving mon data... 2024-04-15T15:18:22.433 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/data/mon.a.tgz 2024-04-15T15:18:22.433 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-04-15T15:18:22.449 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.B6v7SExRYf 2024-04-15T15:18:22.449 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.B6v7SExRYf 2024-04-15T15:18:22.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.B6v7SExRYf 2024-04-15T15:18:22.677 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.B6v7SExRYf is 356KB 2024-04-15T15:18:22.783 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.B6v7SExRYf 2024-04-15T15:18:22.798 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/data/mon.b.tgz 2024-04-15T15:18:22.799 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-04-15T15:18:22.817 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.WswtK66688 2024-04-15T15:18:22.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WswtK66688 2024-04-15T15:18:22.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.WswtK66688 2024-04-15T15:18:23.064 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.WswtK66688 is 479KB 2024-04-15T15:18:23.161 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.WswtK66688 2024-04-15T15:18:23.176 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/data/mon.c.tgz 2024-04-15T15:18:23.176 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-04-15T15:18:23.232 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.pQfXjby7Ip 2024-04-15T15:18:23.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.pQfXjby7Ip 2024-04-15T15:18:23.388 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.pQfXjby7Ip 2024-04-15T15:18:23.475 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.pQfXjby7Ip is 479KB 2024-04-15T15:18:23.572 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.pQfXjby7Ip 2024-04-15T15:18:23.587 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-15T15:18:23.588 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:18:23.590 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:18:23.676 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:18:23.712 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:18:23.713 INFO:tasks.ceph:Archiving crash dumps... 2024-04-15T15:18:23.714 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/remote/smithi046/crash 2024-04-15T15:18:23.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:18:23.748 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/remote/smithi078/crash 2024-04-15T15:18:23.749 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:18:23.779 INFO:tasks.ceph:Shutting down logrotate 2024-04-15T15:18:23.780 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:18:23.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:18:23.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:18:23.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:18:23.936 INFO:tasks.ceph:Compressing logs... 2024-04-15T15:18:23.936 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:18:23.950 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:18:24.032 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-15T15:18:24.032 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-04-15T15:18:24.032 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44755.log 2024-04-15T15:18:24.032 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37754.log 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.1.44755.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37914.log 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37874.log 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37914.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37754.log.gz 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.1.44755.log.gz 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37914.log.gz 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37874.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-15T15:18:24.033 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37874.log.gz 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.45585.log 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.3.45585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45245.log 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37794.log 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.1.45245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43556.log 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37994.log 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37794.log.gz 2024-04-15T15:18:24.034 INFO:teuthology.orchestra.run.smithi078.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.3.45585.log.gz 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.1.43556.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.1.45245.log.gz 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37994.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37994.log.gz 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.7.log: 90.3%gzip -- replaced with /var/log/ceph/ceph-client.1.43556.log.gz -5 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38034.log 2024-04-15T15:18:24.035 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-osd.5.log: --verbose -- /var/log/ceph/ceph-client.admin.37834.log 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38034.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37834.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38034.log.gz 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44013.log 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37834.log.gz 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.3.44013.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.log: 91.4% -- replaced with /var/log/ceph/ceph-client.3.44013.log.gz 2024-04-15T15:18:24.036 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-15T15:18:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.b.log: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:18:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-15T15:18:24.037 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mds.d.log: 91.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-15T15:18:24.046 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.c.log: 91.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-15T15:18:24.087 INFO:teuthology.orchestra.run.smithi078.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-15T15:18:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56758.log 2024-04-15T15:18:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45631.log 2024-04-15T15:18:24.121 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49735.log 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45631.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56758.log.gz 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53230.log 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45631.log.gz 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/ceph-client.admin.49735.log: --verbose -- /var/log/ceph/ceph-client.admin.58065.log 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53230.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49735.log.gz 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53230.log.gz 2024-04-15T15:18:24.122 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43631.log 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58065.log.gz 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48800.log 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58317.log 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43631.log.gz 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58107.log 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58317.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48800.log.gz 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59820.log 2024-04-15T15:18:24.123 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58317.log.gz/var/log/ceph/ceph-client.admin.58107.log: 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54319.log 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58107.log.gz 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46639.log 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51497.log 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59820.log.gz 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46639.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54319.log.gz 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58952.log 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46639.log.gz 2024-04-15T15:18:24.124 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57609.log 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58952.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51497.log.gz 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54587.log 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58952.log.gz 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50785.log 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57609.log.gz 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57567.log 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54587.log.gz 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52623.log 2024-04-15T15:18:24.125 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59087.log 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50785.log.gz 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57567.log.gz 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46471.log 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59087.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52623.log.gz 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46555.log 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46471.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59087.log.gz 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55769.log 2024-04-15T15:18:24.126 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46471.log.gz 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50045.log 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55769.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46555.log.gz 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52085.log 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55769.log.gz 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46807.log 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50045.log.gz 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45211.log 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3%/var/log/ceph/ceph-client.admin.46807.log: -- replaced with /var/log/ceph/ceph-client.admin.52085.log.gz 2024-04-15T15:18:24.127 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45805.log 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45211.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46807.log.gz 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55227.log 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45211.log.gz 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50249.log 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45805.log.gz 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59173.log 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6%/var/log/ceph/ceph-client.admin.50249.log: -- replaced with /var/log/ceph/ceph-client.admin.55227.log.gz 2024-04-15T15:18:24.128 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58588.log 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50249.log.gz 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58419.log 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58588.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59173.log.gz 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48380.log 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58419.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58588.log.gz 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45808.log 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58419.log.gz 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51125.log 2024-04-15T15:18:24.129 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45808.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48380.log.gz 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43248.log 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51125.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45808.log.gz 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51125.log.gz 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56800.log 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43248.log.gz 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46146.log 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gz 2024-04-15T15:18:24.130 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45421.log 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56800.log.gz 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6%/var/log/ceph/ceph-client.admin.45421.log: -- replaced with /var/log/ceph/ceph-client.admin.46146.log.gz 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54461.log 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46723.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45421.log.gz 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43029.log 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2024-04-15T15:18:24.131 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50459.log 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54461.log.gz/var/log/ceph/ceph-client.admin.43029.log: 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45253.log 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43029.log.gz 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59258.log 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50459.log.gz 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50207.log 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59778.log 2024-04-15T15:18:24.132 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45253.log.gz 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50207.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59258.log.gz 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58504.log 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50207.log.gz 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50417.log 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58504.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59778.log.gz 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59594.log 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58504.log.gz 2024-04-15T15:18:24.133 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58868.log 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50417.log.gz 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48968.log 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58868.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.59594.log.gz -5 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53566.log 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58868.log.gz 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49877.log 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48968.log.gz 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53776.log 2024-04-15T15:18:24.134 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49877.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52961.log 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49877.log.gz 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48884.log 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52961.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53776.log.gz 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44965.log 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48884.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52961.log.gz 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.56029.log 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48884.log.gz 2024-04-15T15:18:24.135 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49178.log 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.2.56029.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44965.log.gz 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46144.log 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51455.log 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49178.log.gz 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49346.log 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51455.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46144.log.gz 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50501.log 2024-04-15T15:18:24.136 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51455.log.gz 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46387.log 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49346.log: 91.3% -- replaced with /var/log/ceph/ceph-client.2.56029.log.gz 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53142.log 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49346.log.gz 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46387.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50501.log.gz 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45379.log 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49262.log 2024-04-15T15:18:24.137 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53142.log.gz 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48548.log 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49262.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45379.log.gz 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51875.log 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49262.log.gz 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48926.log 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48548.log.gz 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45463.log 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48926.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51875.log.gz 2024-04-15T15:18:24.138 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59678.log 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48926.log.gz 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58023.log 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45463.log.gz 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59510.log 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58023.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59678.log.gz 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57729.log 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58023.log.gz 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49136.log 2024-04-15T15:18:24.139 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59510.log.gz 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43735.log 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57729.log.gz 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49136.log.gz 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53049.log 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2024-04-15T15:18:24.140 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53049.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43735.log.gz 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48506.log 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53049.log.gz 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57365.log 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58910.log 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48506.log: /var/log/ceph/ceph-client.admin.57365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59636.log 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48506.log.gz 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57365.log.gz 2024-04-15T15:18:24.141 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44823.log 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58910.log.gz 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58462.log 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54036.log 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59636.log.gz 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44823.log.gz 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45812.log 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58462.log.gz 2024-04-15T15:18:24.142 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.54036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55017.log 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56356.log 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55017.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.45812.log.gz 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43987.log 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55017.log.gz 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.0.54036.log.gz 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52539.log 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46597.log 2024-04-15T15:18:24.143 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56356.log.gz 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56993.log 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43987.log.gz 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46597.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52539.log.gz -5 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48270.log 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46597.log.gz 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46765.log 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56993.log.gz 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49094.log 2024-04-15T15:18:24.144 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46765.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48270.log.gz 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58672.log 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46765.log.gz 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51959.log 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49094.log.gz 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45295.log 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51959.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58672.log.gz 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:18:24.145 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51959.log.gz 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48422.log 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45295.log.gz 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59215.log 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52835.log 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48422.log.gz 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59037.log 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59215.log.gz 2024-04-15T15:18:24.146 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57525.log 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52835.log.gz 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57077.log 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59037.log.gz 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51287.log 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57525.log.gz 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57651.log 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57077.log.gz 2024-04-15T15:18:24.147 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44865.log 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57651.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51287.log.gz 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45337.log 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57651.log.gz 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46149.log 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44865.log.gz 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54933.log 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51245.log 2024-04-15T15:18:24.148 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45337.log.gz 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46149.log.gz 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56642.log 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54933.log.gz 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47004.log 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45813.log 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51245.log.gz 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4%/var/log/ceph/ceph-client.admin.47004.log: -- replaced with /var/log/ceph/ceph-client.admin.56642.log.gz 2024-04-15T15:18:24.149 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55604.log 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47004.log.gz 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57465.log 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.55604.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45813.log.gz -5 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50701.log 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57465.log.gz 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54813.log 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43819.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50701.log.gz 2024-04-15T15:18:24.150 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49430.log 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52253.log 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54813.log: /var/log/ceph/ceph-client.admin.49430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56228.log 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54813.log.gz 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52253.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49430.log.gz 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50375.log 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.0.55604.log.gz 2024-04-15T15:18:24.151 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56890.log 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52253.log.gz 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51749.log 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50375.log: /var/log/ceph/ceph-client.admin.56890.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50375.log.gz 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.56228.log.gz 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57939.log 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58546.log 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51749.log: /var/log/ceph/ceph-client.admin.57939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57323.log 2024-04-15T15:18:24.152 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.56890.log.gz -- replaced with /var/log/ceph/ceph-client.admin.51749.log.gz 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58546.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57939.log.gz 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43291.log 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58546.log.gz 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53626.log 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57323.log.gz 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49609.log 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43291.log.gz 2024-04-15T15:18:24.153 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52751.log 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51917.log 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53626.log.gz 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49609.log.gz 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48106.log 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52751.log.gz 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44739.log 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51917.log: /var/log/ceph/ceph-client.2.48106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-04-15T15:18:24.154 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51917.log.gz 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59468.log 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.2.48106.log.gz 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49052.log 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59468.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45007.log 2024-04-15T15:18:24.155 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59468.log.gz 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46513.log 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49052.log.gz 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52127.log 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46513.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45007.log.gz 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54545.log 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46513.log.gz 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54891.log 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52127.log.gz 2024-04-15T15:18:24.156 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57897.log 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54545.log.gz 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54891.log.gz/var/log/ceph/ceph-client.admin.52459.log: 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43333.log 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43333.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-15T15:18:24.157 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57897.log.gz 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43333.log.gz 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51329.log 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50333.log 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55059.log 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50333.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51329.log.gz 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50743.log 2024-04-15T15:18:24.158 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50333.log.gz 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59552.log 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55059.log.gz 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48758.log 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50743.log.gz 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58233.log 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59552.log: /var/log/ceph/ceph-client.admin.48758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51623.log 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59552.log.gz 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48758.log.gz 2024-04-15T15:18:24.159 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52337.log 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37706.log 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51623.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58233.log.gz 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51623.log.gz 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54361.log 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.37706.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46145.log 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52337.log.gz 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54361.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37706.log.gz 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44463.log 2024-04-15T15:18:24.160 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54361.log.gz 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54235.log 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46145.log: /var/log/ceph/ceph-client.admin.44463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47047.log 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46145.log.gz 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53668.log 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54235.log.gz 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49961.log 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53668.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47047.log.gz 2024-04-15T15:18:24.161 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43547.log 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43206.log 89.0% 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53668.log.gz 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.44463.log.gz 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49961.log.gz 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49777.log 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50827.log 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43547.log.gz 2024-04-15T15:18:24.162 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43206.log.gz 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53272.log 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50827.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49777.log.gz 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59384.log 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50827.log.gz 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55727.log 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53272.log.gz 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55101.log 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59384.log.gz 2024-04-15T15:18:24.163 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49693.log 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55727.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55727.log.gz 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50543.log 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43375.log 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49693.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55101.log.gz 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50543.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49693.log.gz 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46962.log 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50543.log.gz 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48674.log 2024-04-15T15:18:24.164 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43375.log: /var/log/ceph/ceph-client.admin.46962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43903.log 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43375.log.gz 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48674.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46962.log.gz 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57239.log 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48674.log.gz 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53398.log 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43903.log.gz 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54193.log 2024-04-15T15:18:24.165 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57239.log: /var/log/ceph/ceph-client.admin.53398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47217.log 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57239.log.gz 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53398.log.gz 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56848.log 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57855.log 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47217.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54193.log.gz 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56848.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47217.log.gz 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45589.log 2024-04-15T15:18:24.166 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56848.log.gz 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54645.log 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57855.log: /var/log/ceph/ceph-client.admin.45589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56702.log 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57855.log.gz 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54645.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45589.log.gz 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59129.log 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54645.log.gz 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2024-04-15T15:18:24.167 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56702.log.gz 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52581.log 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59129.log: /var/log/ceph/ceph-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51833.log 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59129.log.gz 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52581.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51413.log 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52581.log.gz 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50123.log 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51833.log: /var/log/ceph/ceph-client.admin.51413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38059.log 2024-04-15T15:18:24.168 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51833.log.gz 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51413.log.gz 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50123.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50123.log.gz 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48632.log 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.38059.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38059.log.gz 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57407.log 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55269.log 2024-04-15T15:18:24.169 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48632.log: /var/log/ceph/ceph-client.admin.57407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45547.log 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48632.log.gz 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57407.log.gz 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43459.log 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55269.log.gz 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43861.log 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 89.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45547.log.gz /var/log/ceph/ceph-client.0.47608.log 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-15T15:18:24.170 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43459.log: /var/log/ceph/ceph-client.admin.43861.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43459.log.gz 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52919.log 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.47608.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43861.log.gz 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51040.log 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46148.log 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.0.47608.log.gz 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52919.log.gz 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48590.log 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.46148.log: 89.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.51539.log 2024-04-15T15:18:24.171 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51040.log.gz 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48590.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46148.log.gz 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54277.log 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48590.log.gz 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56294.log 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54277.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51539.log.gz 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49835.log 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54277.log.gz 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52379.log 2024-04-15T15:18:24.172 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52211.log 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49835.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56294.log.gz 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51707.log 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49835.log.gz 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52379.log.gz 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44697.log 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45091.log 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52211.log.gz 2024-04-15T15:18:24.173 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44697.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51707.log.gz 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48842.log 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45091.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44697.log.gz 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59426.log 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45091.log.gz 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58826.log 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57813.log 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6%/var/log/ceph/ceph-client.admin.59426.log: -- replaced with /var/log/ceph/ceph-client.admin.48842.log.gz 2024-04-15T15:18:24.174 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45049.log 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59426.log.gz 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58826.log.gz 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50585.log 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53005.log 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57813.log.gz 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45049.log: /var/log/ceph/ceph-client.admin.50585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49567.log 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45049.log.gz 2024-04-15T15:18:24.175 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50585.log.gz 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50998.log 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46681.log 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53005.log.gz 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49567.log: /var/log/ceph/ceph-client.admin.50998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51581.log 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49567.log.gz 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50998.log.gz 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46681.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51791.log 2024-04-15T15:18:24.176 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46681.log.gz 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51581.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58995.log 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51581.log.gz 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53440.log 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58995.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51791.log.gz 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54687.log 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53440.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58995.log.gz 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42948.log 2024-04-15T15:18:24.177 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53440.log.gz 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59300.log 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54687.log.gz 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.42948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50659.log 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42948.log.gz 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57035.log 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50659.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59300.log.gz 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46894.log 2024-04-15T15:18:24.178 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50659.log.gz 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53100.log 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.46894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54975.log 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57035.log.gz 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51371.log 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46894.log.gz 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53100.log.gz 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54503.log 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-15T15:18:24.179 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54975.log: /var/log/ceph/ceph-client.admin.51371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55185.log 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54975.log.gz 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4%/var/log/ceph/ceph-client.admin.54503.log: -- replaced with /var/log/ceph/ceph-client.admin.51371.log.gz 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59738.log 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.55185.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58630.log 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54503.log.gz 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49220.log 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55185.log.gz 2024-04-15T15:18:24.180 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59738.log.gz 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47730.log 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58149.log 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58630.log.gz 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49220.log: /var/log/ceph/ceph-client.admin.47730.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49220.log.gz 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.46429.log 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47730.log.gz 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48228.log 2024-04-15T15:18:24.181 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58149.log.gz 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44923.log 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46429.log.gz 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45505.log 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54771.log 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48228.log.gz 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44923.log: /var/log/ceph/ceph-client.admin.45505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57771.log 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44923.log.gz 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45505.log.gz 2024-04-15T15:18:24.182 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54771.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57281.log 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54771.log.gz 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57771.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43693.log 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57771.log.gz 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43417.log 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57281.log.gz 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43693.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31686.log 2024-04-15T15:18:24.183 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43417.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43693.log.gz 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50291.log 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43417.log.gz 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.31686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51167.log 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31686.log.gz 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43125.log 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51167.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50291.log.gz 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57119.log 2024-04-15T15:18:24.184 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43125.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51167.log.gz 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54419.log 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43125.log.gz 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37956.log 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54419.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57119.log.gz 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52295.log 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.37956.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54419.log.gz 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0%/var/log/ceph/ceph-client.admin.52295.log: -- replaced with /var/log/ceph/ceph-client.admin.37956.log.gz 2024-04-15T15:18:24.185 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52877.log 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52295.log.gz 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43503.log 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52877.log.gz 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr: 84.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.47089.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46135.log 2024-04-15T15:18:24.186 INFO:teuthology.orchestra.run.smithi046.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43503.log.gz 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48464.log 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50932.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46135.log.gz 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52001.log 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-04-15T15:18:24.187 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48464.log: /var/log/ceph/ceph-client.admin.52001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58191.log 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50870.log: 89.5% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48464.log.gz 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52001.log.gz 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53314.log 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.58191.log: -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50003.log 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58191.log.gz 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3%/var/log/ceph/ceph-mds.c.log: -- replaced with /var/log/ceph/ceph-client.admin.53314.log.gz 2024-04-15T15:18:24.188 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56151.log 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50003.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50003.log.gz 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49472.log 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.56151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55344.log 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56151.log.gz 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49472.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49472.log.gz 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53184.log 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2024-04-15T15:18:24.189 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55344.log.gz 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52667.log 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45133.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53184.log.gz 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49651.log 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57161.log 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52667.log.gz 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58275.log 2024-04-15T15:18:24.190 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57161.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49651.log.gz 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57161.log.gz 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53524.log 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46345.log 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58275.log.gz 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55143.log 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53524.log.gz 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46345.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46345.log.gz 2024-04-15T15:18:24.191 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51083.log 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.55143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51665.log 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55143.log.gz 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49304.log 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.51665.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51083.log.gz 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51665.log.gz 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53710.log 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45803.log 2024-04-15T15:18:24.192 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49304.log.gz 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45811.log 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45803.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45803.log.gz 91.1% -- replaced with /var/log/ceph/ceph-client.admin.53710.log.gz 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54729.log 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.45811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44781.log 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45811.log.gz 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.54729.log: gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.48716.log 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54729.log.gz 2024-04-15T15:18:24.193 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44781.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44781.log.gz 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57981.log 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.48716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59342.log 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48716.log.gz 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.57981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53482.log 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.59342.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57981.log.gz 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59342.log.gz 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58359.log 2024-04-15T15:18:24.194 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46147.log 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53482.log.gz 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.58359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46142.log 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3%/var/log/ceph/ceph-client.admin.46147.log: -- replaced with /var/log/ceph/ceph-client.admin.58359.log.gz 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46147.log.gz 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52169.log 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46142.log.gz 2024-04-15T15:18:24.195 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46850.log 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.43945.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52169.log.gz 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53356.log 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.46850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44053.log 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46850.log.gz 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.53356.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53356.log.gz 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44053.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44053.log.gz 2024-04-15T15:18:24.196 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44655.log 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49010.log 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.44655.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44655.log.gz 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52793.log 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49919.log 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49010.log.gz 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52043.log 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.49919.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52793.log.gz 2024-04-15T15:18:24.197 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49919.log.gz 2024-04-15T15:18:24.198 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50165.log 2024-04-15T15:18:24.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.52043.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52043.log.gz 2024-04-15T15:18:24.198 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.50165.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50165.log.gz 2024-04-15T15:18:24.199 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-15T15:18:24.199 INFO:teuthology.orchestra.run.smithi078.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-15T15:18:24.199 INFO:teuthology.orchestra.run.smithi078.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-15T15:18:24.286 INFO:teuthology.orchestra.run.smithi046.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-15T15:18:24.645 INFO:teuthology.orchestra.run.smithi078.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-15T15:18:24.668 INFO:teuthology.orchestra.run.smithi078.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-15T15:18:24.689 INFO:teuthology.orchestra.run.smithi078.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-15T15:18:24.691 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-15T15:18:24.694 INFO:teuthology.orchestra.run.smithi078.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-15T15:18:24.696 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-04-15T15:18:24.696 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m0.698s 2024-04-15T15:18:24.696 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m2.913s 2024-04-15T15:18:24.696 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.138s 2024-04-15T15:18:24.751 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-15T15:18:24.756 INFO:teuthology.orchestra.run.smithi046.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-15T15:18:24.774 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-15T15:18:24.776 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-15T15:18:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m0.814s 2024-04-15T15:18:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m3.237s 2024-04-15T15:18:24.777 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.324s 2024-04-15T15:18:24.777 INFO:tasks.ceph:Archiving logs... 2024-04-15T15:18:24.777 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/remote/smithi046/log 2024-04-15T15:18:24.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:18:25.574 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/remote/smithi078/log 2024-04-15T15:18:25.575 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:18:26.061 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-15T15:18:26.070 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-15T15:18:26.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:18:26.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:18:26.165 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T15:18:26.174 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T15:18:26.175 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:18:26.177 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:18:26.192 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:18:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:18:26.197 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:18:26.197 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-15T15:18:26.197 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +1575us[+1574us] +/- 51ms 2024-04-15T15:18:26.198 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -1813us[-1813us] +/- 77ms 2024-04-15T15:18:26.198 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -649us[ -650us] +/- 74ms 2024-04-15T15:18:26.198 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:18:26.199 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:18:26.199 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2024-04-15T15:18:26.200 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv01.front.sepia.ceph.com 3 6 377 10 +1582us[+1589us] +/- 51ms 2024-04-15T15:18:26.200 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -1798us[-1791us] +/- 77ms 2024-04-15T15:18:26.200 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -639us[ -639us] +/- 74ms 2024-04-15T15:18:26.200 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:18:26.200 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T15:18:26.209 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T15:18:26.210 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T15:18:26.219 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:18:26.323 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:18:26.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:18:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:18:26.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:18:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:18:26.463 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T15:18:26.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T15:18:26.482 INFO:teuthology.task.internal:Duration was 607.127859 seconds 2024-04-15T15:18:26.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T15:18:26.492 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T15:18:26.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:18:26.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:18:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:18:26.548 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:18:26.848 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T15:18:26.849 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-04-15T15:18:26.849 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:18:26.869 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-04-15T15:18:26.870 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:18:26.898 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T15:18:26.898 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:18:26.912 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:18:26.967 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T15:18:26.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:18:26.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:18:27.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T15:18:27.120 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T15:18:27.121 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:18:27.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:18:27.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T15:18:27.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:18:27.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:18:27.215 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-04-15T15:18:27.243 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-04-15T15:18:27.260 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:18:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:18:27.286 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:18:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:18:27.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T15:18:27.325 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T15:18:27.326 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/remote/smithi046 2024-04-15T15:18:27.326 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:18:27.364 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123/remote/smithi078 2024-04-15T15:18:27.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:18:27.402 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T15:18:27.402 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:18:27.405 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:18:27.451 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T15:18:27.468 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T15:18:27.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T15:18:27.484 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T15:18:27.485 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:18:27.488 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:18:27.500 INFO:teuthology.orchestra.run.smithi046.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:18 /home/ubuntu/cephtest 2024-04-15T15:18:27.506 INFO:teuthology.orchestra.run.smithi078.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:18 /home/ubuntu/cephtest 2024-04-15T15:18:27.507 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T15:18:27.518 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T15:18:27.541 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T15:18:27.570 DEBUG:teuthology.nuke:shortname: smithi046 2024-04-15T15:18:27.571 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:18:27.596 DEBUG:teuthology.nuke:shortname: smithi078 2024-04-15T15:18:27.596 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:18:27.602 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:18.559692', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD9vR3gjx85IDEfOdCoscp48OpEZQiPGteXc/kLsFQoKsj1FtDW56a5VP1VFl1/MZbE3p1Oqz2pcwSST7cMdE0Y='} 2024-04-15T15:18:27.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657123', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:18.562129', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWnxiGt5uOpN3O37WMgvVBLr5l137jxAV/h1H3Zhujy7KjZ40v1QMx6J6bXcq6NOBEL8EhbD9c3MroETUQ4HjQ='} 2024-04-15T15:18:27.656 INFO:teuthology.orchestra.console.smithi046:Power off 2024-04-15T15:18:27.656 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:18:27.674 INFO:teuthology.orchestra.console.smithi078:Power off 2024-04-15T15:18:27.674 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:18:27.685 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-04-15T15:18:27.686 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:27.697 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2024-04-15T15:18:27.698 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:27.713 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-15T15:18:27.722 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-04-15T15:18:31.718 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:31.727 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:31.830 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-15T15:18:31.839 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2024-04-15T15:18:35.832 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:35.843 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:35.945 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-04-15T15:18:35.953 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2024-04-15T15:18:36.047 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-04-15T15:18:36.054 INFO:teuthology.orchestra.console.smithi078:Power off completed 2024-04-15T15:18:36.160 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} duration: 607.1278593540192 failure_reason: 'Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=55ccec76d8c44a38bd30e3295474c246 status: fail success: false 2024-04-15T15:18:36.161 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:18:36.244 INFO:teuthology.run:FAIL